Time Series Analysis, Anomaly Detection and Forecasting on GOOGL Stock Price Data¶

Import necessary libreries¶

In [1]:
import numpy as np
import pandas as pd

import seaborn as sns

from pandas import datetime
from datetime import datetime, date 

from statsmodels.graphics.tsaplots import plot_acf
from statsmodels.graphics.tsaplots import plot_pacf
from statsmodels.tsa.arima_process import ArmaProcess
from statsmodels.tsa.arima.model import ARIMA
from statsmodels.tsa.stattools import acf

import matplotlib.pyplot as plt
from matplotlib.dates import DateFormatter
import matplotlib.dates as mdates
from pylab import rcParams

from sklearn.metrics import mean_absolute_error
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score

import pickle
import plotly.express as px
import plotly.graph_objects as go
import random

import math 

plt.style.use('./d4sci.mplstyle')
%matplotlib inline
rcParams['figure.figsize'] = 15, 12
rcParams['axes.labelsize'] = 20
rcParams['ytick.labelsize'] = 16
rcParams['xtick.labelsize'] = 16

import warnings # Supress warnings 
warnings.filterwarnings('ignore')

/var/folders/j4/19v5bck14f94qj1scfkf1m5m0000gn/T/ipykernel_24727/66812729.py:6: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.
  from pandas import datetime

Load the dataset¶

Dataset is taken from Kaggle : https://www.kaggle.com/datasets/varpit94/google-stock-data

In [2]:
df = pd.read_csv("GOOGL.csv", parse_dates=True)
In [3]:
df
Out[3]:
Date Open High Low Close Adj Close Volume
0 2004-08-19 50.050049 52.082081 48.028027 50.220219 50.220219 44659096
1 2004-08-20 50.555557 54.594597 50.300301 54.209209 54.209209 22834343
2 2004-08-23 55.430431 56.796799 54.579578 54.754753 54.754753 18256126
3 2004-08-24 55.675674 55.855858 51.836838 52.487488 52.487488 15247337
4 2004-08-25 52.532532 54.054054 51.991993 53.053055 53.053055 9188602
... ... ... ... ... ... ... ...
4426 2022-03-18 2668.489990 2724.879883 2645.169922 2722.510010 2722.510010 2223100
4427 2022-03-21 2723.270020 2741.000000 2681.850098 2722.030029 2722.030029 1341600
4428 2022-03-22 2722.030029 2821.000000 2722.030029 2797.360107 2797.360107 1774800
4429 2022-03-23 2774.050049 2791.770020 2756.699951 2765.510010 2765.510010 1257700
4430 2022-03-24 2784.000000 2832.379883 2755.010010 2831.439941 2831.439941 1317900

4431 rows × 7 columns

In [4]:
# Converting "Date" column type to datetime64[ns]
df['Date'] = pd.to_datetime(df['Date'])
In [5]:
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4431 entries, 0 to 4430
Data columns (total 7 columns):
 #   Column     Non-Null Count  Dtype         
---  ------     --------------  -----         
 0   Date       4431 non-null   datetime64[ns]
 1   Open       4431 non-null   float64       
 2   High       4431 non-null   float64       
 3   Low        4431 non-null   float64       
 4   Close      4431 non-null   float64       
 5   Adj Close  4431 non-null   float64       
 6   Volume     4431 non-null   int64         
dtypes: datetime64[ns](1), float64(5), int64(1)
memory usage: 242.4 KB
In [6]:
df.isnull().any()
Out[6]:
Date         False
Open         False
High         False
Low          False
Close        False
Adj Close    False
Volume       False
dtype: bool

There is no null value in any column

Data Visualization¶

In [7]:
# Plotting the graph of "Open" field of the dataset

data = [
    go.Scatter(x=df['Date'], y=df['Open'], name='Open'),
    go.Scatter(x=df['Date'], y=df['Close'], name='Close'),
]

fig = go.Figure(data)

fig.update_layout(
    title="Google sock price train",
    xaxis_title="Time",
    yaxis_title="Price",
    xaxis=dict(rangeslider=dict(visible=True))
    )


fig.show()
In [8]:
corr = df.corr()
corr.style.background_gradient(cmap = 'coolwarm')
Out[8]:
  Open High Low Close Adj Close Volume
Open 1.000000 0.999915 0.999902 0.999808 0.999808 -0.453884
High 0.999915 1.000000 0.999877 0.999903 0.999903 -0.452855
Low 0.999902 0.999877 1.000000 0.999914 0.999914 -0.455447
Close 0.999808 0.999903 0.999914 1.000000 1.000000 -0.454252
Adj Close 0.999808 0.999903 0.999914 1.000000 1.000000 -0.454252
Volume -0.453884 -0.452855 -0.455447 -0.454252 -0.454252 1.000000

According to the correlation table Open, High, Low, Close and Adj Close are correlated/totally dependent on each other so we can drop High, Low, Close and Adj Close. For further analysis we will only use "Open" value

Running Values¶

In [9]:
# Plotting the graph of "Open" field of the dataset

data = [
    go.Scatter(x=df['Date'], y=df['Open'], name='Open'),
    go.Scatter(x=df['Date'], y=df['Open'].rolling(window= 30).mean(), name='Open rolling 30 days'),
    go.Scatter(x=df['Date'], y=df['Open'].rolling(window= 60).mean(), name='Open rolling 60 days'),
]

fig = go.Figure(data)

fig.update_layout(
    title="Google sock price train",
    xaxis_title="Time",
    yaxis_title="Price",
    xaxis=dict(rangeslider=dict(visible=True))
    )

fig.show()

Resampling¶

This time series data was collected as per days. For the better analysis and to get additional information from data we have done Resampling.

Resampling can be done 2 ways.

  1. Upsampling : Converting data to larger scale. Eg. From days to week or month
  2. Downsampling : Converting data to smaller scale. Eg. From week to days

Here, we have done upsampling of open price

In [10]:
resampled_df_7D = df[['Date','Open']].resample('7D', on='Date').sum().reset_index(drop=False)
resampled_df_14D = df[['Date','Open']].resample('15D', on='Date').sum().reset_index(drop=False)

data = [
    go.Scatter(x=df['Date'], y=df['Open'], name='Daily Open Price'),
    go.Scatter(x=resampled_df_7D['Date'], y=resampled_df_7D['Open'], name='Weekly Open Price'),
    go.Scatter(x=resampled_df_14D['Date'], y=resampled_df_14D['Open'], name='Bi-Weekly Open Price'),
]

fig = go.Figure(data)

fig.update_layout(
    title="Resampled",
    xaxis_title="Time",
    yaxis_title="Price",
    xaxis=dict(rangeslider=dict(visible=True))
    )

fig.show()

Bollinger Bands¶

For the better bollinger bands, we have resampled the data first to bi-weekly and plotted the upper and lower bound of bollinger bands

In [11]:
# As we can see earlier, downsample to weekly could smooth the data and help with analysis
downsample = df[['Date',
                 'Open', 
                ]].resample('14D', on='Date').mean().reset_index(drop=False)

df_copy = downsample.copy()
In [12]:
size = 20
std_dev = 2
df_copy['mov_avg'] = df_copy['Open'].rolling(window=size).mean() 
df_copy['std_dev'] = df_copy['Open'].rolling(window=size).std()  
df_copy['BB_up'] = df_copy['mov_avg'] + (df_copy['std_dev'] * std_dev)
df_copy['BB_low'] = df_copy['mov_avg'] - (df_copy['std_dev'] * std_dev)
In [13]:
data = [
    go.Scatter(x=df_copy['Date'], y=df_copy['Open'], name='Open'),
    go.Scatter(x=df_copy['Date'], y=df_copy['BB_up'], name='Upper Band'),
    go.Scatter(x=df_copy['Date'], y=df_copy['BB_low'], name='Lower Band'),
]

fig = go.Figure(data)

fig.update_layout(
    title="Bollinger Band",
    xaxis_title="Time",
    yaxis_title="Price",
    xaxis=dict(rangeslider=dict(visible=True))
    )

fig.show()

Stationarity Check¶

There are 2 checks done in order to determine whether open price data is stationary or not,

  1. Rolling Statistics
  2. Augmented Dickey Fuller Test

1. Rolling Statistics¶

In [14]:
from datetime import date 

rolling_window = 52
f, ax = plt.subplots(nrows=1, ncols=1, figsize=(15, 12))

sns.lineplot(x=df['Date'], y=df['Open'], ax=ax, color='dodgerblue')
sns.lineplot(x=df['Date'], y=df['Open'].rolling(rolling_window).mean(), ax=ax, color='black', label='rolling mean')
sns.lineplot(x=df['Date'], y=df['Open'].rolling(rolling_window).std(), ax=ax, color='red', label='rolling std')

plt.show()

In the plot for Open price and Volume, it can be seen that mean and standard deviation is not constant, so clearly there the series is non-stationary.

2. Augmented Dickey Fuller Test¶

Augmented Dickey-Fuller (ADF) test is a statistical test also known as unit root test. Unit roots are a reason for non-stationarity.

Null Hypothesis (H0): Time series has a unit root then time series is not stationary.

Alternate Hypothesis (H1): Time series has no unit root then time series is stationary.

  1. the null hypothesis can be rejected if the p-value is below a set significance level. The defaults significance level is 5%

    p-value > significance level (default: 0.05): Fail to reject the null hypothesis (H0), the data has a unit root and is non-stationary.

    p-value <= significance level (default: 0.05): Reject the null hypothesis (H0), the data does not have a unit root and is stationary.

  1. the null hypothesis can be rejects if the test statistic is less than the critical value.

    ADF statistic > critical value: Fail to reject the null hypothesis (H0), the data has a unit root and is non-stationary.

    ADF statistic < critical value: Reject the null hypothesis (H0), the data does not have a unit root and is stationary.

In [15]:
from statsmodels.tsa.stattools import adfuller
result = adfuller(df['Open'].values)
result

ouput_series = pd.Series(result[0:4], index = ['Test Statistics', 'P-value', 'No. Of Lags Used', 'No. Of Observation Used'])
for index, value in result[4].items():
    ouput_series['Critical Value %s'%index] = value
    
print(ouput_series)

Test Statistics               3.106417
P-value                       1.000000
No. Of Lags Used             31.000000
No. Of Observation Used    4399.000000
Critical Value 1%            -3.431837
Critical Value 5%            -2.862197
Critical Value 10%           -2.567120
dtype: float64

So as we can see from the Dickey Fuller Test Statistics that p-value is very large compare to significance level which is 0.5. So we fail to reject the null hypothesis, and can determine that the data has a unit root and is non-stationary. Also, the ADF statistics is greater than critical value at(5%), Fail to reject the null hypothesis (H0), the data has a unit root and is non-stationary.

Stationarize the series¶

Now to make time series stationary, there are 2 ways

  1. Transformation : Log, Square, Squareroot to stabilize non constant variance
  2. Differencing : Subtract current value from previous one

1. Transformation¶

In [16]:
df['Open_Log'] = np.log(abs(df['Open']))

f, ax = plt.subplots(figsize=(20, 6))
sns.lineplot(x=df['Date'], y=df['Open_Log'], color='dodgerblue')
sns.lineplot(x=df['Date'], y=df['Open_Log'].rolling(rolling_window).mean(), color='black', label='rolling mean')
sns.lineplot(x=df['Date'], y=df['Open_Log'].rolling(rolling_window).std(), color='red', label='rolling std')

# We will perform ADF
result = adfuller(df['Open_Log'].values)
result

print("Dicky Fuller Test Statistics")
ouput_series = pd.Series(result[0:4], index = ['Test Statistics', 'P-value', 'No. Of Lags Used', 'No. Of Observation Used'])
for index, value in result[4].items():
    ouput_series['Critical Value %s'%index] = value
    
print(ouput_series)

Dicky Fuller Test Statistics
Test Statistics              -1.619239
P-value                       0.473128
No. Of Lags Used              1.000000
No. Of Observation Used    4429.000000
Critical Value 1%            -3.431827
Critical Value 5%            -2.862193
Critical Value 10%           -2.567117
dtype: float64

Rolling statistics observation : The plot for log transformation value have better result than the previous one but the mean and standard deviation is not constant it's varying with time, so time series is non-stationary.

Augmented Dickey Fuller Test : The p-value is still greater than 0.5, so fail reject the null hypothesis and the series is non-stationary.

2. Differencing¶

In [17]:
rolling_window = 52
diff = df['Open'].diff(1).dropna()
df['Open_diff'] = np.append([0], diff)

f, ax = plt.subplots(figsize=(15, 6))

sns.lineplot(x=df['Date'], y=df['Open_diff'], color='dodgerblue')
sns.lineplot(x=df['Date'], y=df['Open_diff'].rolling(rolling_window).mean(), color='black', label='rolling mean')
sns.lineplot(x=df['Date'], y=df['Open_diff'].rolling(rolling_window).std(), color='red', label='rolling std')

result = adfuller(df['Open_diff'].values)
print("Dicky Fuller Test Statistics")
ouput_series = pd.Series(result[0:4], index = ['Test Statistics', 'P-value', 'No. Of Lags Used', 'No. Of Observation Used'])
for index, value in result[4].items():
    ouput_series['Critical Value %s'%index] = value
    
print(ouput_series)

Dicky Fuller Test Statistics
Test Statistics           -1.220843e+01
P-value                    1.178880e-22
No. Of Lags Used           3.100000e+01
No. Of Observation Used    4.399000e+03
Critical Value 1%         -3.431837e+00
Critical Value 5%         -2.862197e+00
Critical Value 10%        -2.567120e+00
dtype: float64

Rolling statistics observation : The plot for log transformation value have better result than the previous one but the mean and standard deviation is not constant it's varying with time, so time series is non-stationary.

Augmented Dickey Fuller Test : The p-value is now less than 0.5, so fail reject the null hypothesis, the data does not have a unit root and is stationary

Components of Time Series¶

We have taken a sample from the time series data to visualize the trend, seasonality and residuals components of time series

In [18]:
from statsmodels.tsa.seasonal import seasonal_decompose

decomposition = seasonal_decompose(df['Open'].iloc[::30],
                                   model='additive',
                                   period = 52)
decomposition.plot()
plt.show()

We have plotted the graph for time series component Trend, Season, Residuals of Open_diff value to check whether seasonal component is stationary or not after difference.

In [19]:
from statsmodels.tsa.seasonal import seasonal_decompose

decomposition = seasonal_decompose(df['Open_diff'],
                                   model='additive',
                                   period = 52)
decomposition.plot()
plt.show()

As per the plot, seasonal component is stationary so there is no seasonality in time series

Auto Correlation Plot¶

In [20]:
plot_acf(df['Open'].diff(1).dropna(), lags=105);

Skipping the first lag (at 0 on the X-axis), we see that there is one point outside of the blue highlighted region. This means that we have one significant lag. Thus, we will use q = 1.

Partial Auto Correlation Plot¶

In [21]:
plot_pacf(df['Open'].diff(1).dropna(), lags=20);

Skipping the first lag (at 0 on the X-axis), we see that there is one point outside of the blue highlighted region. This means that we have one significant lag. Thus, we will use p = 1.

Split Dataset¶

In [22]:
copy_df = df[['Open']]

size = int(len(copy_df) * 0.90)

train, test = copy_df[0:size], copy_df[size:len(copy_df)]

print(len(train), len(test))

3987 444

Time Series Models¶

This univariate time series models we are going to implement for the given series¶

  • AR Model
  • MA Model
  • ARMA Model
  • ARIMA Model
  • Auto-ARIMA Model
  • Simple Exponential Smooting
  • Holt-Winter's Exponential Smooting

1. AR (Auto Regressive) Model:¶

According to Partial Auto Correlation Plot, order of p term for AR model is 1.

In [23]:
from statsmodels.tsa.ar_model import AutoReg

ar_model = AutoReg(train['Open'], lags=1).fit()
ar_model.summary()
Out[23]:
AutoReg Model Results
Dep. Variable: Open No. Observations: 3987
Model: AutoReg(1) Log Likelihood -15120.203
Method: Conditional MLE S.D. of innovations 10.744
Date: Sun, 26 Jun 2022 AIC 30246.406
Time: 20:10:48 BIC 30265.277
Sample: 1 HQIC 30253.096
3987
coef std err z P>|z| [0.025 0.975]
const 0.2695 0.295 0.914 0.361 -0.309 0.848
Open.L1 1.0002 0.000 2159.585 0.000 0.999 1.001
Roots
Real Imaginary Modulus Frequency
AR.1 0.9998 +0.0000j 0.9998 0.0000
In [24]:
ar_pred = ar_model.predict(start=len(train), end=(len(train) + len(test)-1), dynamic=False)

plt.plot(test, color='red', label = 'Actual')
plt.plot(ar_pred, color = 'blue', label = 'Forecast')
plt.title("Auto Regressive Model Forecasting")
plt.legend()
plt.show()
In [25]:
MAE = mean_absolute_error(test, ar_pred)
MSE = mean_squared_error(test, ar_pred)
RMSE = math.sqrt(MSE)
R2 = r2_score(test, ar_pred)

print("Mean Absolute Error:", MAE)
print("Mean Squared Error:", MSE)
print("Root Mean Squared Error:", RMSE)
print("R Squared:", R2)

Mean Absolute Error: 692.7789631072875
Mean Squared Error: 689765.0696627054
Root Mean Squared Error: 830.5209628075052
R Squared: -1.5311243293547685

2. MA(Moving Average) Model¶

According to Auto Correlation Plot, order of q term for MA model is 1.

In [26]:
from statsmodels.tsa.arima.model import ARIMA

ma_model = ARIMA(train, order=(0, 0, 1)).fit()
ma_model.summary()
Out[26]:
SARIMAX Results
Dep. Variable: Open No. Observations: 3987
Model: ARIMA(0, 0, 1) Log Likelihood -26518.860
Date: Sun, 26 Jun 2022 AIC 53043.720
Time: 20:10:49 BIC 53062.592
Sample: 0 HQIC 53050.411
- 3987
Covariance Type: opg
coef std err z P>|z| [0.025 0.975]
const 521.8239 8.223 63.459 0.000 505.707 537.941
ma.L1 0.9847 0.002 412.463 0.000 0.980 0.989
sigma2 3.504e+04 1228.805 28.515 0.000 3.26e+04 3.74e+04
Ljung-Box (L1) (Q): 3680.81 Jarque-Bera (JB): 549.82
Prob(Q): 0.00 Prob(JB): 0.00
Heteroskedasticity (H): 2.49 Skew: 0.89
Prob(H) (two-sided): 0.00 Kurtosis: 2.62


Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
In [27]:
ma_pred = ma_model.predict(start=len(train), end=(len(df)-1), dynamic=False)

plt.plot(test, color='red', label = 'Actual')
plt.plot(ma_pred, color = 'blue', label = 'Forecast')
plt.title("Moving Average Model Forecasting")
plt.legend()
plt.show()
In [28]:
MAE = mean_absolute_error(test, ma_pred)
MSE = mean_squared_error(test, ma_pred)
RMSE = math.sqrt(MSE)
R2 = r2_score(test, ma_pred)

print("Mean Absolute Error:", MAE)
print("Mean Squared Error:", MSE)
print("Root Mean Squared Error:", RMSE)
print("R Squared:", R2)

Mean Absolute Error: 1719.0427810254334
Mean Squared Error: 3229621.634704502
Root Mean Squared Error: 1797.1148084372633
R Squared: -10.85124363895151

3. ARMA (Auto Regressive Moving Average) Model¶

In [29]:
model = ARIMA(train['Open'], order = (2,0,2))
arma_model = model.fit()
In [30]:
arma_pred = arma_model.predict(start=len(train), end=(len(df)-1), dynamic=False, alpha = 0)

plt.plot(test, color ='red', label = 'Actual')
plt.plot(arma_pred, color = 'blue', label = 'Forecast')
plt.title("Auto Regressive Moving Average Model Forecasting")
plt.legend()
plt.show()
In [31]:
MAE = mean_absolute_error(test, arma_pred)
MSE = mean_squared_error(test, arma_pred)
RMSE = math.sqrt(MSE)
R2 = r2_score(test, arma_pred)

print("Mean Absolute Error:", MAE)
print("Mean Squared Error:", MSE)
print("Root Mean Squared Error:", RMSE)
print("R Squared:", R2)

Mean Absolute Error: 826.0738522531162
Mean Squared Error: 966780.1179738705
Root Mean Squared Error: 983.2497739505819
R Squared: -2.547643662119246

4. ARIMA Model¶

In [32]:
model = ARIMA(train, order=(5, 3, 3))
model_fit = model.fit()
model_fit.summary()
Out[32]:
SARIMAX Results
Dep. Variable: Open No. Observations: 3987
Model: ARIMA(5, 3, 3) Log Likelihood -15472.965
Date: Sun, 26 Jun 2022 AIC 30963.930
Time: 20:10:53 BIC 31020.540
Sample: 0 HQIC 30984.001
- 3987
Covariance Type: opg
coef std err z P>|z| [0.025 0.975]
ar.L1 -1.3477 0.088 -15.303 0.000 -1.520 -1.175
ar.L2 -0.8231 0.064 -12.796 0.000 -0.949 -0.697
ar.L3 -0.7492 0.055 -13.606 0.000 -0.857 -0.641
ar.L4 -0.5659 0.046 -12.367 0.000 -0.656 -0.476
ar.L5 -0.2059 0.028 -7.299 0.000 -0.261 -0.151
ma.L1 -0.4909 0.088 -5.577 0.000 -0.663 -0.318
ma.L2 -0.8732 0.106 -8.245 0.000 -1.081 -0.666
ma.L3 0.3649 0.032 11.477 0.000 0.303 0.427
sigma2 155.6400 1.543 100.851 0.000 152.615 158.665
Ljung-Box (L1) (Q): 41.18 Jarque-Bera (JB): 49054.10
Prob(Q): 0.00 Prob(JB): 0.00
Heteroskedasticity (H): 12.29 Skew: 0.38
Prob(H) (two-sided): 0.00 Kurtosis: 20.17


Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
In [33]:
arima_pred = model_fit.predict(start=len(train), end=(len(df)-1), dynamic=False)

plt.plot(test, color ='red', label = 'Actual')
plt.plot(arima_pred, color = 'blue', label = 'Forecast')
plt.title("Auto Regressive Integrated Moving Average Model Forecasting")
plt.legend()
plt.show()
In [34]:
MAE = mean_absolute_error(test, arima_pred)
MSE = mean_squared_error(test, arima_pred)
RMSE = math.sqrt(MSE)
R2 = r2_score(test, arima_pred)

print("Mean Absolute Error:", MAE)
print("Mean Squared Error:", MSE)
print("Root Mean Squared Error:", RMSE)
print("R Squared:", R2)

Mean Absolute Error: 338.4243511274503
Mean Squared Error: 177924.7985074022
Root Mean Squared Error: 421.81133046351687
R Squared: 0.3470968506453198

5. Auto ARIMA¶

In [35]:
pip install pmdarima

Requirement already satisfied: pmdarima in /Users/vrunda/miniforge3/lib/python3.9/site-packages (1.8.5)
Requirement already satisfied: joblib>=0.11 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pmdarima) (1.1.0)
Requirement already satisfied: Cython!=0.29.18,>=0.29 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pmdarima) (0.29.30)
Requirement already satisfied: numpy>=1.19.3 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pmdarima) (1.21.6)
Requirement already satisfied: pandas>=0.19 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pmdarima) (1.4.2)
Requirement already satisfied: scikit-learn>=0.22 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pmdarima) (1.0.2)
Requirement already satisfied: scipy>=1.3.2 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pmdarima) (1.8.0)
Requirement already satisfied: statsmodels!=0.12.0,>=0.11 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pmdarima) (0.13.2)
Requirement already satisfied: urllib3 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pmdarima) (1.26.9)
Requirement already satisfied: setuptools!=50.0.0,>=38.6.0 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pmdarima) (60.10.0)
Requirement already satisfied: python-dateutil>=2.8.1 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pandas>=0.19->pmdarima) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from pandas>=0.19->pmdarima) (2022.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from scikit-learn>=0.22->pmdarima) (3.1.0)
Requirement already satisfied: packaging>=21.3 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from statsmodels!=0.12.0,>=0.11->pmdarima) (21.3)
Requirement already satisfied: patsy>=0.5.2 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from statsmodels!=0.12.0,>=0.11->pmdarima) (0.5.2)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from packaging>=21.3->statsmodels!=0.12.0,>=0.11->pmdarima) (3.0.7)
Requirement already satisfied: six in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from patsy>=0.5.2->statsmodels!=0.12.0,>=0.11->pmdarima) (1.15.0)
Note: you may need to restart the kernel to use updated packages.
In [36]:
from pmdarima import auto_arima 

stepwise_fit = auto_arima(df['Open'], trace=True, suppress_warnings=True)
stepwise_fit.summary()

Performing stepwise search to minimize aic
 ARIMA(2,2,2)(0,0,0)[0]             : AIC=inf, Time=0.62 sec
 ARIMA(0,2,0)(0,0,0)[0]             : AIC=40865.476, Time=0.03 sec
 ARIMA(1,2,0)(0,0,0)[0]             : AIC=39277.349, Time=0.03 sec
 ARIMA(0,2,1)(0,0,0)[0]             : AIC=inf, Time=0.17 sec
 ARIMA(2,2,0)(0,0,0)[0]             : AIC=38839.822, Time=0.05 sec
 ARIMA(3,2,0)(0,0,0)[0]             : AIC=38572.234, Time=0.19 sec
 ARIMA(4,2,0)(0,0,0)[0]             : AIC=38333.960, Time=0.28 sec
 ARIMA(5,2,0)(0,0,0)[0]             : AIC=38182.202, Time=0.31 sec
 ARIMA(5,2,1)(0,0,0)[0]             : AIC=inf, Time=1.48 sec
 ARIMA(4,2,1)(0,0,0)[0]             : AIC=inf, Time=1.05 sec
 ARIMA(5,2,0)(0,0,0)[0] intercept   : AIC=38184.196, Time=0.42 sec

Best model:  ARIMA(5,2,0)(0,0,0)[0]          
Total fit time: 4.630 seconds
Out[36]:
SARIMAX Results
Dep. Variable: y No. Observations: 4431
Model: SARIMAX(5, 2, 0) Log Likelihood -19085.101
Date: Sun, 26 Jun 2022 AIC 38182.202
Time: 20:10:59 BIC 38220.578
Sample: 0 HQIC 38195.735
- 4431
Covariance Type: opg
coef std err z P>|z| [0.025 0.975]
ar.L1 -0.8912 0.005 -183.127 0.000 -0.901 -0.882
ar.L2 -0.6712 0.007 -93.805 0.000 -0.685 -0.657
ar.L3 -0.5347 0.007 -80.584 0.000 -0.548 -0.522
ar.L4 -0.3865 0.008 -47.045 0.000 -0.403 -0.370
ar.L5 -0.1848 0.007 -28.222 0.000 -0.198 -0.172
sigma2 324.1476 1.975 164.123 0.000 320.277 328.019
Ljung-Box (L1) (Q): 3.09 Jarque-Bera (JB): 119124.55
Prob(Q): 0.08 Prob(JB): 0.00
Heteroskedasticity (H): 32.56 Skew: 0.83
Prob(H) (two-sided): 0.00 Kurtosis: 28.35


Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
In [37]:
auto_model = ARIMA(train,order=(5,2,0) )
auto_model = auto_model.fit()
auto_model.summary()
Out[37]:
SARIMAX Results
Dep. Variable: Open No. Observations: 3987
Model: ARIMA(5, 2, 0) Log Likelihood -15426.964
Date: Sun, 26 Jun 2022 AIC 30865.928
Time: 20:10:59 BIC 30903.670
Sample: 0 HQIC 30879.309
- 3987
Covariance Type: opg
coef std err z P>|z| [0.025 0.975]
ar.L1 -0.9037 0.006 -150.331 0.000 -0.915 -0.892
ar.L2 -0.7125 0.009 -80.930 0.000 -0.730 -0.695
ar.L3 -0.5040 0.010 -49.359 0.000 -0.524 -0.484
ar.L4 -0.3632 0.010 -38.100 0.000 -0.382 -0.344
ar.L5 -0.1682 0.007 -23.786 0.000 -0.182 -0.154
sigma2 134.8742 1.030 130.971 0.000 132.856 136.893
Ljung-Box (L1) (Q): 4.03 Jarque-Bera (JB): 43440.36
Prob(Q): 0.04 Prob(JB): 0.00
Heteroskedasticity (H): 12.38 Skew: 0.35
Prob(H) (two-sided): 0.00 Kurtosis: 19.16


Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
In [38]:
auto_pred = auto_model.predict(start=len(train), end=(len(df)-1), typ='levels')

plt.plot(test, color ='red', label = 'Actual')
plt.plot(auto_pred, color = 'blue', label = 'Forecast')
plt.title("Auto ARIMA Model Forecasting")
plt.legend()
plt.show()
In [39]:
MAE = mean_absolute_error(test, auto_pred)
MSE = mean_squared_error(test, auto_pred)
RMSE = math.sqrt(MSE)
R2 = r2_score(test, auto_pred)

print("Mean Absolute Error:", MAE)
print("Mean Squared Error:", MSE)
print("Root Mean Squared Error:", RMSE)
print("R Squared:", R2)

Mean Absolute Error: 537.6490020546744
Mean Squared Error: 426047.58884201315
Root Mean Squared Error: 652.7232099764901
R Squared: -0.5634010259584228

6. Simple Exponential Smoothing¶

In [40]:
from statsmodels.tsa.holtwinters import SimpleExpSmoothing

alpha = 0.1
es = []
es.append(df.at[0,'Open'])

def exponetial_smooting(x):
    zt = alpha * df.at[x,'Open'] + (1 - alpha) * es[x-1]
    es.append(round(zt, 1))
    
for i in range(1,len(df)):
    exponetial_smooting(i)

df['Open_pred'] = es

df[['Open','Open_pred']].plot(title = 'Exponential Smoothing')
Out[40]:
<AxesSubplot:title={'center':'Exponential Smoothing'}>

7. Holt Winter's Exponential Smoothing¶

In [41]:
from statsmodels.tsa.holtwinters import ExponentialSmoothing

model = ExponentialSmoothing(train['Open'], trend = 'mul', seasonal = 'mul', seasonal_periods = 12).fit()

hw_pred = model.forecast(len(test))
plt.plot(test, color = 'red', label = 'Actual')
plt.plot(hw_pred, color = 'blue', label = 'Forecast')
plt.title('Holt Winters Triple Exponential Smoothing')
Out[41]:
Text(0.5, 1.0, 'Holt Winters Triple Exponential Smoothing')
In [42]:
MAE = mean_absolute_error(test, hw_pred)
MSE = mean_squared_error(test, hw_pred)
RMSE = math.sqrt(MSE)
R2 = r2_score(test, hw_pred)

print("Mean Absolute Error:", MAE)
print("Mean Squared Error:", MSE)
print("Root Mean Squared Error:", RMSE)
print("R Squared:", R2)

Mean Absolute Error: 477.6993834523344
Mean Squared Error: 336528.6227645655
Root Mean Squared Error: 580.1108710966944
R Squared: -0.2349071039798698

8. Recurrent Neural Network (RNN)¶

In [43]:
df_train = pd.read_csv('Google_train.csv',parse_dates=True)
In [44]:
training_set=df['Open']
training_set=pd.DataFrame(training_set)
In [45]:
dataset_test = pd.read_csv('Google_test_with_anomaly.csv',index_col="Date",parse_dates=True)
In [46]:
# Feature Scaling
from sklearn.preprocessing import MinMaxScaler

sc = MinMaxScaler(feature_range = (0, 1))
training_set_scaled = sc.fit_transform(training_set)
In [47]:
# Creating a structure with 60 timesteps abd a output based on 60 data values
X_train = []
y_train = []
for i in range(60, 4131):
    X_train.append(training_set_scaled[i-60:i, 0])
    y_train.append(training_set_scaled[i, 0])
X_train, y_train = np.array(X_train), np.array(y_train)

print("X_train : ",X_train)
print("y_train : ",y_train)

# Reshaping
X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))

X_train :  [[1.36253641e-04 3.06152003e-04 1.94456941e-03 ... 1.26010132e-02
  1.20240333e-02 1.17649829e-02]
 [3.06152003e-04 1.94456941e-03 2.02699418e-03 ... 1.20240333e-02
  1.17649829e-02 1.44732507e-02]
 [1.94456941e-03 2.02699418e-03 9.70602048e-04 ... 1.17649829e-02
  1.44732507e-02 1.36691804e-02]
 ...
 [5.02412376e-01 5.08236876e-01 5.12975839e-01 ... 5.63668928e-01
  5.80608090e-01 5.80561032e-01]
 [5.08236876e-01 5.12975839e-01 4.95223314e-01 ... 5.80608090e-01
  5.80561032e-01 5.69920274e-01]
 [5.12975839e-01 4.95223314e-01 5.10932359e-01 ... 5.80561032e-01
  5.69920274e-01 5.63857120e-01]]
y_train :  [0.01447325 0.01366918 0.01317294 ... 0.56992027 0.56385712 0.57109997]

Part 2 - Building the RNN

In [48]:
# Importing the libraries
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Dropout
In [49]:
# Initialise the basic neural network(Sequential)
regressor = Sequential()

Metal device set to: Apple M1

2022-06-26 20:11:04.440277: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2022-06-26 20:11:04.440526: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
In [50]:
# First layer with 20% dropout
regressor.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 1)))
regressor.add(Dropout(0.2))

# Second layer
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

# Third layer
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

# Fourth layer
regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.2))

# Adding the final output layer
regressor.add(Dense(units = 1))
In [51]:
# Compiling RNN with "ADAM" optimizer
regressor.compile(optimizer = 'adam', loss = 'mean_squared_error')

# Fitting the RNN to the Training set of the data
regressor.fit(X_train, y_train, epochs = 100, batch_size = 32)

Epoch 1/100

2022-06-26 20:11:04.872542: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2022-06-26 20:11:06.133333: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:11:06.505712: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:11:06.621906: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:11:06.708022: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:11:06.800421: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:11:06.943768: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:11:07.122058: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:11:07.273840: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:11:07.419532: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.

128/128 [==============================] - 10s 56ms/step - loss: 0.0021
Epoch 2/100
128/128 [==============================] - 7s 52ms/step - loss: 2.7246e-04
Epoch 3/100
128/128 [==============================] - 7s 52ms/step - loss: 2.2894e-04
Epoch 4/100
128/128 [==============================] - 7s 51ms/step - loss: 1.7314e-04
Epoch 5/100
128/128 [==============================] - 7s 53ms/step - loss: 1.6791e-04
Epoch 6/100
128/128 [==============================] - 7s 52ms/step - loss: 1.6695e-04
Epoch 7/100
128/128 [==============================] - 7s 52ms/step - loss: 1.5289e-04
Epoch 8/100
128/128 [==============================] - 7s 54ms/step - loss: 1.4558e-04
Epoch 9/100
128/128 [==============================] - 7s 54ms/step - loss: 1.2979e-04
Epoch 10/100
128/128 [==============================] - 7s 52ms/step - loss: 1.9844e-04
Epoch 11/100
128/128 [==============================] - 7s 53ms/step - loss: 1.3960e-04
Epoch 12/100
128/128 [==============================] - 7s 52ms/step - loss: 1.6761e-04
Epoch 13/100
128/128 [==============================] - 7s 53ms/step - loss: 1.0929e-04
Epoch 14/100
128/128 [==============================] - 7s 51ms/step - loss: 1.0806e-04
Epoch 15/100
128/128 [==============================] - 7s 52ms/step - loss: 1.0515e-04
Epoch 16/100
128/128 [==============================] - 6s 51ms/step - loss: 9.5926e-05
Epoch 17/100
128/128 [==============================] - 7s 52ms/step - loss: 8.8509e-05
Epoch 18/100
128/128 [==============================] - 7s 53ms/step - loss: 9.5650e-05
Epoch 19/100
128/128 [==============================] - 7s 52ms/step - loss: 1.0539e-04
Epoch 20/100
128/128 [==============================] - 7s 53ms/step - loss: 1.0465e-04
Epoch 21/100
128/128 [==============================] - 7s 52ms/step - loss: 7.9479e-05
Epoch 22/100
128/128 [==============================] - 7s 54ms/step - loss: 8.1866e-05
Epoch 23/100
128/128 [==============================] - 7s 53ms/step - loss: 8.0082e-05
Epoch 24/100
128/128 [==============================] - 7s 55ms/step - loss: 8.1798e-05
Epoch 25/100
128/128 [==============================] - 7s 54ms/step - loss: 7.3234e-05
Epoch 26/100
128/128 [==============================] - 7s 54ms/step - loss: 9.3877e-05
Epoch 27/100
128/128 [==============================] - 7s 53ms/step - loss: 7.6107e-05
Epoch 28/100
128/128 [==============================] - 7s 53ms/step - loss: 6.9415e-05
Epoch 29/100
128/128 [==============================] - 7s 53ms/step - loss: 7.0430e-05
Epoch 30/100
128/128 [==============================] - 7s 52ms/step - loss: 9.8971e-05
Epoch 31/100
128/128 [==============================] - 7s 53ms/step - loss: 8.0870e-05
Epoch 32/100
128/128 [==============================] - 7s 53ms/step - loss: 6.6623e-05
Epoch 33/100
128/128 [==============================] - 7s 53ms/step - loss: 6.5206e-05
Epoch 34/100
128/128 [==============================] - 7s 52ms/step - loss: 7.1278e-05
Epoch 35/100
128/128 [==============================] - 7s 52ms/step - loss: 6.3132e-05
Epoch 36/100
128/128 [==============================] - 7s 52ms/step - loss: 5.7117e-05
Epoch 37/100
128/128 [==============================] - 7s 52ms/step - loss: 6.5922e-05
Epoch 38/100
128/128 [==============================] - 7s 53ms/step - loss: 6.1787e-05
Epoch 39/100
128/128 [==============================] - 7s 53ms/step - loss: 5.7993e-05
Epoch 40/100
128/128 [==============================] - 7s 52ms/step - loss: 6.3152e-05
Epoch 41/100
128/128 [==============================] - 7s 52ms/step - loss: 6.0408e-05
Epoch 42/100
128/128 [==============================] - 7s 52ms/step - loss: 5.3150e-05
Epoch 43/100
128/128 [==============================] - 7s 52ms/step - loss: 5.1629e-05
Epoch 44/100
128/128 [==============================] - 7s 53ms/step - loss: 5.8112e-05
Epoch 45/100
128/128 [==============================] - 7s 53ms/step - loss: 5.0175e-05
Epoch 46/100
128/128 [==============================] - 7s 52ms/step - loss: 5.4696e-05
Epoch 47/100
128/128 [==============================] - 7s 54ms/step - loss: 4.8781e-05
Epoch 48/100
128/128 [==============================] - 7s 52ms/step - loss: 4.4388e-05
Epoch 49/100
128/128 [==============================] - 7s 52ms/step - loss: 4.4142e-05
Epoch 50/100
128/128 [==============================] - 7s 52ms/step - loss: 6.6922e-05
Epoch 51/100
128/128 [==============================] - 7s 52ms/step - loss: 4.4923e-05
Epoch 52/100
128/128 [==============================] - 7s 54ms/step - loss: 4.5479e-05
Epoch 53/100
128/128 [==============================] - 7s 56ms/step - loss: 5.1943e-05
Epoch 54/100
128/128 [==============================] - 7s 55ms/step - loss: 5.1328e-05
Epoch 55/100
128/128 [==============================] - 7s 55ms/step - loss: 4.1149e-05
Epoch 56/100
128/128 [==============================] - 7s 54ms/step - loss: 4.0908e-05
Epoch 57/100
128/128 [==============================] - 7s 55ms/step - loss: 4.0702e-05
Epoch 58/100
128/128 [==============================] - 8s 65ms/step - loss: 4.4244e-05
Epoch 59/100
128/128 [==============================] - 8s 62ms/step - loss: 4.2203e-05
Epoch 60/100
128/128 [==============================] - 8s 59ms/step - loss: 3.6515e-05
Epoch 61/100
128/128 [==============================] - 7s 58ms/step - loss: 3.7393e-05
Epoch 62/100
128/128 [==============================] - 8s 59ms/step - loss: 4.5367e-05
Epoch 63/100
128/128 [==============================] - 7s 56ms/step - loss: 3.9695e-05
Epoch 64/100
128/128 [==============================] - 7s 54ms/step - loss: 4.2579e-05
Epoch 65/100
128/128 [==============================] - 7s 55ms/step - loss: 3.7059e-05
Epoch 66/100
128/128 [==============================] - 7s 55ms/step - loss: 3.7120e-05
Epoch 67/100
128/128 [==============================] - 7s 55ms/step - loss: 3.8323e-05
Epoch 68/100
128/128 [==============================] - 7s 55ms/step - loss: 3.4141e-05
Epoch 69/100
128/128 [==============================] - 7s 55ms/step - loss: 3.9991e-05
Epoch 70/100
128/128 [==============================] - 7s 54ms/step - loss: 3.4205e-05
Epoch 71/100
128/128 [==============================] - 7s 55ms/step - loss: 3.2171e-05
Epoch 72/100
128/128 [==============================] - 7s 55ms/step - loss: 3.5252e-05
Epoch 73/100
128/128 [==============================] - 7s 55ms/step - loss: 4.4552e-05
Epoch 74/100
128/128 [==============================] - 7s 55ms/step - loss: 4.0026e-05
Epoch 75/100
128/128 [==============================] - 7s 54ms/step - loss: 3.9700e-05
Epoch 76/100
128/128 [==============================] - 7s 56ms/step - loss: 3.4830e-05
Epoch 77/100
128/128 [==============================] - 8s 62ms/step - loss: 3.5148e-05
Epoch 78/100
128/128 [==============================] - 8s 61ms/step - loss: 4.3178e-05
Epoch 79/100
128/128 [==============================] - 7s 57ms/step - loss: 3.3276e-05
Epoch 80/100
128/128 [==============================] - 7s 56ms/step - loss: 3.1875e-05
Epoch 81/100
128/128 [==============================] - 8s 64ms/step - loss: 3.5506e-05
Epoch 82/100
128/128 [==============================] - 9s 68ms/step - loss: 3.2672e-05
Epoch 83/100
128/128 [==============================] - 7s 58ms/step - loss: 4.0423e-05
Epoch 84/100
128/128 [==============================] - 7s 58ms/step - loss: 3.4441e-05
Epoch 85/100
128/128 [==============================] - 7s 55ms/step - loss: 3.9361e-05
Epoch 86/100
128/128 [==============================] - 7s 56ms/step - loss: 3.3639e-05
Epoch 87/100
128/128 [==============================] - 7s 56ms/step - loss: 3.5856e-05
Epoch 88/100
128/128 [==============================] - 7s 54ms/step - loss: 3.8942e-05
Epoch 89/100
128/128 [==============================] - 7s 56ms/step - loss: 3.2837e-05
Epoch 90/100
128/128 [==============================] - 7s 56ms/step - loss: 3.4424e-05
Epoch 91/100
128/128 [==============================] - 7s 56ms/step - loss: 2.7409e-05
Epoch 92/100
128/128 [==============================] - 7s 56ms/step - loss: 3.5277e-05
Epoch 93/100
128/128 [==============================] - 7s 55ms/step - loss: 3.4504e-05
Epoch 94/100
128/128 [==============================] - 7s 54ms/step - loss: 2.7909e-05
Epoch 95/100
128/128 [==============================] - 7s 54ms/step - loss: 2.9908e-05
Epoch 96/100
128/128 [==============================] - 7s 55ms/step - loss: 2.8449e-05
Epoch 97/100
128/128 [==============================] - 7s 54ms/step - loss: 2.9322e-05
Epoch 98/100
128/128 [==============================] - 7s 54ms/step - loss: 2.8452e-05
Epoch 99/100
128/128 [==============================] - 7s 53ms/step - loss: 3.2736e-05
Epoch 100/100
128/128 [==============================] - 7s 53ms/step - loss: 3.5125e-05
Out[51]:
<keras.callbacks.History at 0x2960bd4f0>
In [52]:
# Getting the real values of the data
real_stock_price = dataset_test.iloc[:, 1:2].values
len(real_stock_price)
Out[52]:
300
In [53]:
dataset_test
Out[53]:
Unnamed: 0 Open High Low Close Adj Close Volume
Date
2021-01-15 0 1729.020020 1747.619995 1711.709961 1727.619995 1727.619995 1572200
2021-01-19 1 1743.000000 1803.839966 1732.829956 1784.469971 1784.469971 2083000
2021-01-20 2 1826.369995 1899.719971 1819.510010 1880.069946 1880.069946 3207600
2021-01-21 3 1893.650024 1932.079956 1879.410034 1884.150024 1884.150024 2931700
2021-01-22 4 1891.459961 1904.390015 1876.500000 1892.560059 1892.560059 1637200
... ... ... ... ... ... ... ...
2022-03-18 295 2668.489990 2724.879883 2645.169922 2722.510010 2722.510010 2223100
2022-03-21 296 2723.270020 2741.000000 2681.850098 2722.030029 2722.030029 1341600
2022-03-22 297 2722.030029 2821.000000 2722.030029 2797.360107 2797.360107 1774800
2022-03-23 298 2774.050049 2791.770020 2756.699951 2765.510010 2765.510010 1257700
2022-03-24 299 2784.000000 2832.379883 2755.010010 2831.439941 2831.439941 1317900

300 rows × 7 columns

In [54]:
test_set=dataset_test['Open']
test_set=pd.DataFrame(test_set)
len(test_set)
Out[54]:
300
In [55]:
# Getting the predicted stock price from our model

dataset_total = pd.concat((test['Open'], test['Open']), axis = 0)
inputs = dataset_total[len(dataset_total) - len(dataset_test) - 60:].values
inputs = inputs.reshape(-1,1)
inputs = sc.transform(inputs)
X_test = []
for i in range(60, 300):
    X_test.append(inputs[i-60:i, 0])

print("Len :", len(X_test))
X_test = np.array(X_test)
X_test = np.reshape(X_test, (X_test.shape[0], X_test.shape[1], 1))
# predicted_stock_price = regressor.predict(X_test)
predicted_stock_price = regressor.predict(X_test)

predicted_stock_price = sc.inverse_transform(predicted_stock_price)

Len : 240

2022-06-26 20:22:45.534113: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:22:45.677626: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:22:45.848250: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:22:45.925723: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-06-26 20:22:46.010031: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
In [56]:
predicted_stock_price=pd.DataFrame(predicted_stock_price)
predicted_stock_price.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 240 entries, 0 to 239
Data columns (total 1 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   0       240 non-null    float32
dtypes: float32(1)
memory usage: 1.1 KB
In [57]:
len(predicted_stock_price)
Out[57]:
240
In [58]:
plt.plot(real_stock_price, color = 'red', label = 'Real Google Stock Price')
Out[58]:
[<matplotlib.lines.Line2D at 0x2d5aecc10>]
In [59]:
# Visualising the results of Actual price vs predicted price

plt.plot(real_stock_price, color = 'red', label = 'Real Google Stock Price')
plt.plot(predicted_stock_price, color = 'blue', label = 'Predicted Google Stock Price')
plt.title('Google Stock Price Prediction')
plt.xlabel('Time')
plt.ylabel('Google Stock Price')
plt.legend()
plt.show()
In [60]:
temp_df = predicted_stock_price
temp_df.columns = ["Open"]

temp_df
Out[60]:
Open
0 1655.063721
1 1643.375732
2 1650.643066
3 1714.537231
4 1773.723511
... ...
235 2569.608887
236 2610.753662
237 2616.925049
238 2666.411377
239 2668.881836

240 rows × 1 columns

In [61]:
lst = list(predicted_stock_price['Open'])
real_stock_price_lst = real_stock_price.flatten().tolist()
In [62]:
difference = []
for i in range(len(predicted_stock_price)):
    difference.append(abs(real_stock_price_lst[i] - lst[i]))
lst_2 = sorted(difference, key=float)
print(lst_2[-5:])
print("Anomalies Found :", len(lst_2[-5:]))

[1096.6212158203125, 1447.903076171875, 1632.700927734375, 1993.41064453125, 2364.846435546875]
Anomalies Found : 5

9. Prophet¶

In [63]:
df = pd.read_csv('Google_train.csv',parse_dates=True)

df.drop(["High", "Low", "Close", "Volume", "Adj Close"], axis=1, inplace=True)
df.head()
Out[63]:
Date Open
0 2004-08-19 50.050049
1 2004-08-20 50.555557
2 2004-08-23 55.430431
3 2004-08-24 55.675674
4 2004-08-25 52.532532
In [64]:
df.columns = ["ds", "y"]
In [65]:
!pip install fbprophet

Requirement already satisfied: fbprophet in /Users/vrunda/miniforge3/lib/python3.9/site-packages (0.7.1)
Requirement already satisfied: tqdm>=4.36.1 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (4.63.0)
Requirement already satisfied: convertdate>=2.1.2 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (2.4.0)
Requirement already satisfied: numpy>=1.15.4 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (1.21.6)
Requirement already satisfied: Cython>=0.22 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (0.29.30)
Requirement already satisfied: holidays>=0.10.2 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (0.14.2)
Requirement already satisfied: setuptools-git>=1.2 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (1.2)
Requirement already satisfied: python-dateutil>=2.8.0 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (2.8.2)
Requirement already satisfied: matplotlib>=2.0.0 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (3.5.1)
Requirement already satisfied: pystan>=2.14 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (2.19.1.1)
Requirement already satisfied: pandas>=1.0.4 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (1.4.2)
Requirement already satisfied: cmdstanpy==0.9.5 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (0.9.5)
Requirement already satisfied: LunarCalendar>=0.0.9 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from fbprophet) (0.0.9)
Requirement already satisfied: pymeeus<=1,>=0.3.13 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from convertdate>=2.1.2->fbprophet) (0.5.11)
Requirement already satisfied: hijri-converter in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from holidays>=0.10.2->fbprophet) (2.2.4)
Requirement already satisfied: korean-lunar-calendar in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from holidays>=0.10.2->fbprophet) (0.2.1)
Requirement already satisfied: pytz in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from LunarCalendar>=0.0.9->fbprophet) (2022.1)
Requirement already satisfied: ephem>=3.7.5.3 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from LunarCalendar>=0.0.9->fbprophet) (4.1.3)
Requirement already satisfied: fonttools>=4.22.0 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from matplotlib>=2.0.0->fbprophet) (4.31.2)
Requirement already satisfied: pillow>=6.2.0 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from matplotlib>=2.0.0->fbprophet) (9.1.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from matplotlib>=2.0.0->fbprophet) (1.4.2)
Requirement already satisfied: cycler>=0.10 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from matplotlib>=2.0.0->fbprophet) (0.11.0)
Requirement already satisfied: pyparsing>=2.2.1 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from matplotlib>=2.0.0->fbprophet) (3.0.7)
Requirement already satisfied: packaging>=20.0 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from matplotlib>=2.0.0->fbprophet) (21.3)
Requirement already satisfied: six>=1.5 in /Users/vrunda/miniforge3/lib/python3.9/site-packages (from python-dateutil>=2.8.0->fbprophet) (1.15.0)
In [66]:
from fbprophet import Prophet
In [67]:
model = Prophet()
In [68]:
model.fit(df)

INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

Initial log joint probability = -98.1837
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       12087.8    0.00738753       823.716      0.7964      0.7964      125   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       12186.8   0.000737706       401.332           1           1      240   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       12212.9     0.0208422       1838.98           1           1      353   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       12235.7   0.000465613       142.474           1           1      464   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       12242.9    0.00468064       1117.68      0.3313      0.3313      578   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       12253.6    0.00257326       255.338           1           1      694   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       12258.3    0.00387329       204.884           1           1      814   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       12259.6     0.0037484       351.178           1           1      940   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       12263.1   0.000488029       98.5072           1           1     1047   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     999       12264.4    0.00304397       482.545           1           1     1168   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099       12265.2   0.000896284       87.6571      0.4118           1     1288   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1103       12265.2   2.09643e-05       102.868   6.882e-08       0.001     1332  LS failed, Hessian reset 
    1199       12265.4   7.23206e-05       62.7128       0.344           1     1464   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       12266.1    0.00408731       132.875           1           1     1581   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1399       12266.5    0.00420785       182.737           1           1     1705   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1411       12266.6   9.94623e-06       75.1719   1.175e-07       0.001     1758  LS failed, Hessian reset 
    1499       12266.7   0.000399033       220.284      0.4857           1     1881   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes
Out[68]:
<fbprophet.forecaster.Prophet at 0x174f85370>
   1599       12267.1    0.00752668       238.131           1           1     2002   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1699       12267.3   0.000372186       135.364      0.2763      0.2763     2131   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1701       12267.3   8.98757e-06       55.7635   7.391e-08       0.001     2178  LS failed, Hessian reset 
    1732       12267.4   8.92503e-06       78.5237   1.408e-07       0.001     2264  LS failed, Hessian reset 
    1757       12267.4   2.15373e-06       53.1457      0.5851      0.5851     2296   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
In [69]:
model.component_modes
Out[69]:
{'additive': ['yearly',
  'weekly',
  'additive_terms',
  'extra_regressors_additive',
  'holidays'],
 'multiplicative': ['multiplicative_terms', 'extra_regressors_multiplicative']}
In [70]:
prediction_dates = model.make_future_dataframe(periods= 400)
In [71]:
prediction_dates.tail()
Out[71]:
ds
4526 2022-02-14
4527 2022-02-15
4528 2022-02-16
4529 2022-02-17
4530 2022-02-18
In [72]:
prediction = model.predict(prediction_dates)
In [73]:
prediction.head()
Out[73]:
ds trend yhat_lower yhat_upper trend_lower trend_upper additive_terms additive_terms_lower additive_terms_upper weekly weekly_lower weekly_upper yearly yearly_lower yearly_upper multiplicative_terms multiplicative_terms_lower multiplicative_terms_upper yhat
0 2004-08-19 54.893773 -11.941793 128.503119 54.893773 54.893773 3.270105 3.270105 3.270105 1.595389 1.595389 1.595389 1.674716 1.674716 1.674716 0.0 0.0 0.0 58.163878
1 2004-08-20 55.151020 -12.541342 130.693915 55.151020 55.151020 2.625661 2.625661 2.625661 0.993757 0.993757 0.993757 1.631904 1.631904 1.631904 0.0 0.0 0.0 57.776681
2 2004-08-23 55.922761 -15.729714 129.808859 55.922761 55.922761 1.744965 1.744965 1.744965 0.188017 0.188017 0.188017 1.556949 1.556949 1.556949 0.0 0.0 0.0 57.667726
3 2004-08-24 56.180008 -12.976617 127.034138 56.180008 56.180008 2.469176 2.469176 2.469176 0.942723 0.942723 0.942723 1.526453 1.526453 1.526453 0.0 0.0 0.0 58.649184
4 2004-08-25 56.437255 -15.093815 125.457100 56.437255 56.437255 2.869408 2.869408 2.869408 1.386877 1.386877 1.386877 1.482530 1.482530 1.482530 0.0 0.0 0.0 59.306662
In [74]:
model.plot(prediction)
Out[74]:
In [75]:
model.plot_components(prediction)
Out[75]:
In [76]:
from fbprophet.diagnostics import cross_validation

df_cross_validation = cross_validation(model, initial = "1000 days", period="182 days", horizon="365 days")

INFO:fbprophet:Making 26 forecasts with cutoffs between 2007-08-01 00:00:00 and 2020-01-15 00:00:00

  0%|          | 0/26 [00:00<?, ?it/s]
Initial log joint probability = -6.10668
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       2283.38     0.0110158       1069.79      0.3053           1      126   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     144       2293.91   6.47102e-05       125.235   3.726e-07       0.001      226  LS failed, Hessian reset 
     163       2296.58   0.000552138       352.577    5.74e-06       0.001      281  LS failed, Hessian reset 
     199       2299.37     0.0651433       1297.98           1           1      317   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       2304.31   0.000278361       363.123     0.09587    0.009587      451   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       2306.19   4.91879e-05       116.362      0.8091      0.8091      575   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     464       2307.53   8.10036e-05       140.254   3.134e-07       0.001      734  LS failed, Hessian reset 
     499       2308.62   0.000391613       91.2601           1           1      779   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     516       2308.76   6.21465e-05       117.412   3.133e-07       0.001      844  LS failed, Hessian reset 
     599       2309.06    2.5126e-07       56.0578      0.7596      0.7596      962   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     606       2309.06   4.59654e-07       63.9405       2.277      0.2277      973   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -6.6868
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       2615.27    0.00357644       772.405     0.03092           1      128   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       2625.37     0.0133094       283.385      0.9993      0.9993      241   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       2634.44   0.000250258       105.186           1           1      361   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       2637.14    0.00303074       185.539      0.4517           1      485   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     434       2638.08   5.49032e-05       98.7053   3.392e-07       0.001      568  LS failed, Hessian reset 
     471       2638.69     0.0003722       175.888   2.927e-06       0.001      653  LS failed, Hessian reset 
     499       2638.88    0.00137215       87.5457      0.6951           1      691   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     535       2639.13   7.49418e-05       130.783   3.247e-07       0.001      778  LS failed, Hessian reset 
     599       2639.32    0.00340466       104.733           1           1      857   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     606       2639.75   5.95104e-05       121.458   4.399e-07       0.001      909  LS failed, Hessian reset 
     699       2640.44   3.30954e-05       78.3402           1           1     1025   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     713       2640.49   0.000214095       241.467   3.038e-06       0.001     1084  LS failed, Hessian reset 
     769       2640.54   1.33182e-05       86.7211   2.178e-07       0.001     1204  LS failed, Hessian reset 
     799       2640.54   4.14771e-07       67.3246      0.1805           1     1240   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     802       2640.54   3.03358e-07       64.7488       0.397       0.397     1244   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -16.6835
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       2692.07    0.00486824       110.625      0.3372           1      121   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       2715.17    0.00327795       659.775      0.4093      0.4093      239   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     271       2728.75   0.000315617       171.191   2.589e-06       0.001      373  LS failed, Hessian reset 
     299       2729.44   0.000235369        175.16      0.3914      0.9789      414   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     325       2729.83   0.000111635       131.991   1.115e-06       0.001      482  LS failed, Hessian reset 
     399       2730.97    0.00943941       225.878           1           1      583   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       2732.82    0.00226002       104.428       1.202      0.1752      709   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     589       2733.15   6.26142e-05       74.7436   4.566e-07       0.001      885  LS failed, Hessian reset 
     599       2733.27   0.000245548       69.1353           1           1      897   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     645       2733.42   2.52256e-05        62.064   3.775e-07       0.001      995  LS failed, Hessian reset 
     665       2733.42   1.33951e-07       57.0426      0.3869           1     1035   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -46.1063
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       2737.98    0.00521962        170.47           1           1      121   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       2756.26     0.0256259         147.6           1           1      235   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     252       2767.01   0.000672765       99.3521   3.496e-06       0.001      332  LS failed, Hessian reset 
     299       2769.94   0.000446455       93.0591       0.916       0.916      394   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       2773.98     0.0018786       108.231      0.6465      0.6465      514   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     427       2775.52   0.000912336       215.212    4.56e-06       0.001      600  LS failed, Hessian reset 
     499       2777.44   8.25212e-05       84.2268       0.469       0.469      703   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     516       2777.76   0.000214656       176.058   1.954e-06       0.001      770  LS failed, Hessian reset 
     599       2778.24    0.00353032       68.3863           1           1      867   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       2782.38    0.00216824       188.029           1           1      985   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     797       2783.99   0.000362643       184.728   5.063e-06       0.001     1155  LS failed, Hessian reset 
     799       2784.01   0.000181385        80.722           1           1     1157   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     871       2784.44   0.000355914       87.9872   4.609e-06       0.001     1294  LS failed, Hessian reset 
     899       2784.45   1.90042e-05       57.3611       1.982      0.5004     1337   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     923       2784.47   8.82108e-05       77.3254   9.435e-07       0.001     1401  LS failed, Hessian reset 
     944       2784.47   3.03375e-07       56.0119      0.1649           1     1434   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -34.7148
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       2759.53     0.0127436       338.282           1           1      120   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       2790.81     0.0172421       252.998           1           1      239   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299        2798.3    0.00159888       113.814           1           1      371   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     372       2804.52    0.00036963       205.353   9.725e-07       0.001      505  LS failed, Hessian reset 
     399        2808.4    0.00177134       111.142      0.5331      0.5331      535   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       2811.25   0.000227468       76.5964      0.6182      0.6182      662   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599        2814.1    0.00194286       77.1211      0.3155           1      789   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       2816.01     0.0123172       256.122       1.998      0.1998      908   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     713       2817.52   0.000260441       150.989   8.338e-07       0.001      980  LS failed, Hessian reset 
     799       2819.86   0.000632225       72.7632           1           1     1085   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     821       2820.16   0.000173125       104.643    1.71e-06       0.001     1151  LS failed, Hessian reset 
     899       2820.63   3.29945e-05       59.4669           1           1     1242   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     914       2820.67   0.000107133       111.386    1.09e-06       0.001     1309  LS failed, Hessian reset 
     999       2820.73   4.65748e-06       75.3197      0.3352           1     1420   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1082       2820.74   4.11725e-06       90.0938   5.008e-08       0.001     1570  LS failed, Hessian reset 
    1095       2820.74   7.08026e-08       74.4551      0.2631      0.8436     1585   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -28.0618
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99        3187.9     0.0157831       497.734      0.9111      0.9111      121   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       3373.32    0.00365436       682.178           1           1      234   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       3429.15     0.0366094       813.488      0.3563       0.705      350   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399        3476.7     0.0112365       1027.07      0.4583      0.4583      471   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     452        3484.1   6.50326e-05       91.2704   4.238e-07       0.001      580  LS failed, Hessian reset 
     499       3485.64    0.00119115       369.227           1           1      642   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       3488.56    0.00548162       113.017           1           1      760   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     615       3489.39   7.38862e-05       114.491   4.622e-07       0.001      832  LS failed, Hessian reset 
     699       3490.33    0.00205625       99.9424           1           1      944   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     774       3490.99   0.000215761       75.4137   2.912e-06       0.001     1077  LS failed, Hessian reset 
     799          3491   6.38792e-06       62.7095           1           1     1112   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     882       3493.91   0.000105204       102.762   3.389e-07       0.001     1275  LS failed, Hessian reset 
     899       3494.99    0.00116129       107.516           1           1     1296   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     999       3496.66   0.000211561       108.262      0.1991           1     1429   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1037       3496.88   0.000391364       200.267   4.435e-06       0.001     1519  LS failed, Hessian reset 
    1067       3496.93   6.38652e-05       70.5576   9.213e-07       0.001     1605  LS failed, Hessian reset 
    1099       3496.93   3.97406e-06       83.3432           1           1     1648   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1122       3497.12   0.000142127       204.952   8.328e-07       0.001     1721  LS failed, Hessian reset 
    1199       3497.36   2.34978e-06       58.2213           1           1     1828   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1209       3497.36   5.75609e-08       51.1001      0.1531      0.1531     1842   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -37.4653
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       3243.47    0.00241881       179.237      0.7473      0.7473      113   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       3375.35      0.108554       334.807       2.721      0.2721      226   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       3422.47     0.0430787        492.86           1           1      346   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     387       3456.33   8.72829e-05       89.5839   6.445e-07       0.001      489  LS failed, Hessian reset 
     399       3457.48    0.00362259       157.948           1           1      505   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499        3468.7     0.0032788       406.043      0.1904           1      626   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       3479.72     0.0143177       274.788           1           1      737   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     632       3483.89   0.000193879        208.71    1.21e-06       0.001      825  LS failed, Hessian reset 
     699       3486.74    0.00140625       65.5414       0.401           1      914   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     790       3489.55   0.000100743       119.011   6.795e-07       0.001     1057  LS failed, Hessian reset 
     799       3489.91    0.00111342       162.389           1           1     1071   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     818       3489.97   0.000101796       116.403   9.928e-07       0.001     1133  LS failed, Hessian reset 
     836       3490.01   0.000107677       102.197   1.456e-06       0.001     1193  LS failed, Hessian reset 
     870       3490.03   2.37434e-07       50.8689      0.5373      0.5373     1241   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -29.1152
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       3602.98     0.0526513        892.98      0.8225      0.8225      115   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       3748.93     0.0360412       988.731     0.09457      0.8241      230   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       3821.43     0.0169627       173.719           1           1      341   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       3858.02     0.0373295       1477.64           1           1      457   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       3874.15      0.012575       148.205           1           1      579   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     561       3886.79    0.00012248       125.616   4.304e-07       0.001      690  LS failed, Hessian reset 
     599       3892.36    0.00194392       131.492      0.3726      0.3726      737   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     655       3894.52    0.00011081       141.694   5.001e-07       0.001      842  LS failed, Hessian reset 
     699        3896.3    0.00161904       252.766      0.8329      0.8329      894   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     795       3935.51    0.00246373       819.521   1.449e-05       0.001     1048  LS failed, Hessian reset 
     799       3936.17    0.00102456       176.943           1           1     1052   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       3951.29   0.000463106       75.1691           1           1     1182   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     982       3955.05    9.7735e-05       138.904    7.24e-07       0.001     1329  LS failed, Hessian reset 
     999       3955.75    0.00371675       204.479           1           1     1353   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099        3958.5    0.00102371       71.9626      0.1854           1     1477   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1100       3958.52    0.00039308       159.022   5.462e-06       0.001     1520  LS failed, Hessian reset 
    1197       3961.71   0.000208722       218.514   1.123e-06       0.001     1675  LS failed, Hessian reset 
    1199       3961.73   0.000111561       76.5661           1           1     1677   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       3963.51    0.00109752          98.9      0.6603           1     1803   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1329        3963.7     0.0001905       201.179   1.368e-06       0.001     1886  LS failed, Hessian reset 
    1399       3963.88   5.99434e-05       82.2246       0.377      0.8199     1982   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1429       3964.08   8.42091e-05       94.8464   3.537e-07       0.001     2062  LS failed, Hessian reset 
    1482       3964.18   0.000116323       62.5222   1.488e-06       0.001     2177  LS failed, Hessian reset 
    1499       3964.18   2.26767e-06       73.3085      0.3019      0.3019     2200   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1519       3964.18   7.58616e-07       52.0786   1.143e-08       0.001     2266  LS failed, Hessian reset 
    1535       3964.18   1.52011e-07       51.2348     0.07242           1     2290   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -32.4238
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       3867.21     0.0374807       1099.93           1           1      115   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       3986.77    0.00371253       670.956      0.1154           1      226   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       4033.69     0.0162689       563.424           1           1      338   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       4064.95    0.00697388       547.316           1           1      449   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       4086.94   0.000196552       153.408      0.3962      0.3962      567   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     554       4091.18    0.00018327       217.617   8.792e-07       0.001      702  LS failed, Hessian reset 
     599       4094.28     0.0027324       115.106       4.044      0.8331      755   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       4098.13     0.0027055       149.573           1           1      879   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     778       4099.18   0.000126961       143.051   1.564e-06       0.001     1019  LS failed, Hessian reset 
     799       4099.21   3.11444e-05       53.6989      0.6549      0.6549     1047   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       4099.78    0.00542408       166.213           1           1     1175   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     986       4100.22   0.000270442       212.184   3.503e-06       0.001     1330  LS failed, Hessian reset 
     999       4100.27   0.000612608        99.637           1           1     1345   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1097        4100.4   0.000116803       121.497   1.739e-06       0.001     1520  LS failed, Hessian reset 
    1099       4100.41    6.2966e-05       73.0135           1           1     1522   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1168       4100.44   6.13447e-05       98.4213   7.227e-07       0.001     1659  LS failed, Hessian reset 
    1191       4100.44   2.41279e-07       58.4375           1           1     1695   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -40.7416
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       4076.55    0.00759069       288.168           1           1      117   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       4267.38      0.021588       660.499           1           1      232   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       4360.25     0.0077574       174.452           1           1      343   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       4391.91    0.00411629       195.203           1           1      465   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     447       4409.23    0.00096799       437.928   2.177e-06       0.001      556  LS failed, Hessian reset 
     499       4423.41    0.00367611       601.169       0.153       0.153      625   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       4429.22     0.0143847       287.507           1           1      749   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     619       4430.91   6.65616e-05       60.9674   3.583e-07       0.001      825  LS failed, Hessian reset 
     699       4432.56     0.0208132       307.682           1           1      928   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     737        4442.1   0.000384944       328.054   3.742e-06       0.001     1013  LS failed, Hessian reset 
     799        4450.4   0.000210009       177.156   2.189e-06       0.001     1126  LS failed, Hessian reset 
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     832       4451.21   0.000271597       216.091   3.037e-06       0.001     1210  LS failed, Hessian reset 
     899        4452.3    0.00351611       254.866           1           1     1287   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     920        4452.8   0.000102154       124.674   4.034e-07       0.001     1352  LS failed, Hessian reset 
     999       4454.06    0.00406587       127.367           1           1     1448   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099       4458.45   0.000242096       77.5077           1           1     1578   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1113       4458.63   6.78119e-05       84.8717   4.127e-07       0.001     1633  LS failed, Hessian reset 
    1120       4458.74   7.47211e-05       58.0069   3.352e-07       0.001     1680  LS failed, Hessian reset 
    1163       4458.94    7.0057e-05       96.6228   1.083e-06       0.001     1774  LS failed, Hessian reset 
    1199       4458.98   0.000357937       75.2268      0.3487      0.3487     1824   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       4462.19    0.00130818       128.517      0.7478      0.7478     1938   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1327        4462.4   4.52932e-05       69.4049   4.728e-07       0.001     2033  LS failed, Hessian reset 
    1399       4462.98    0.00136405       100.189      0.8606      0.8606     2119   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1499       4464.39   5.78422e-05       51.8293      0.4602           1     2246   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1538       4464.72   0.000602853       148.185   5.213e-06       0.001     2350  LS failed, Hessian reset 
    1599       4465.15   0.000374805       50.4379           1           1     2426   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1633       4465.21   6.10613e-06       52.4604   1.092e-07       0.001     2511  LS failed, Hessian reset 
    1699       4465.35   0.000233459       115.974           1           1     2595   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1772       4465.58   4.29009e-05       81.4283   7.341e-07       0.001     2739  LS failed, Hessian reset 
    1799       4465.58   1.24932e-05       70.2978      0.4378      0.7719     2778   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1899       4465.72    7.2841e-07       49.3526      0.1464      0.4202     2907   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1908       4465.72   8.86226e-07       46.6394           1           1     2921   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -41.0122
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       4380.88    0.00868454       762.246           1           1      116   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       4570.28     0.0972156       548.069           1           1      227   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       4654.92      0.110634       864.844           1           1      342   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       4692.33     0.0190044       403.873           1           1      457   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       4733.53     0.0028864       425.398      0.8409      0.8409      577   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       4755.08    0.00181321       96.1806      0.4041      0.4041      693   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     686       4761.09   8.07298e-05       114.905   4.046e-07       0.001      833  LS failed, Hessian reset 
     699       4762.14   0.000346138       83.8542           1           1      849   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       4764.91      0.003495        326.02      0.9122      0.9122      966   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       4774.74    0.00156011       217.951           1           1     1079   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     999       4777.68    0.00417527       200.955      0.2982      0.7439     1199   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099       4781.35    0.00161939       211.763       0.301           1     1322   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1199       4783.41   0.000177784       156.158       0.448       0.448     1446   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1254       4783.91   8.01577e-05       137.846   5.685e-07       0.001     1552  LS failed, Hessian reset 
    1299       4784.28   2.69461e-05       56.1678      0.1923      0.1923     1605   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1399       4787.84     0.0289875       203.852           1           1     1721   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1423        4793.8   0.000133363       206.643   5.819e-07       0.001     1830  LS failed, Hessian reset 
    1499       4799.63   0.000435712       128.957      0.4096      0.4096     1916   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1575       4801.29   0.000104211       135.963   8.364e-07       0.001     2057  LS failed, Hessian reset 
    1599       4801.66   0.000694657       112.398     0.03181           1     2094   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1621       4801.77   5.66938e-05       88.6744   7.773e-07       0.001     2166  LS failed, Hessian reset 
    1699       4802.04   0.000426662       134.203      0.5914      0.5914     2252   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1722       4802.31    9.3342e-05       151.373   8.486e-07       0.001     2327  LS failed, Hessian reset 
    1759       4802.45   9.88257e-05       170.047   8.578e-07       0.001     2432  LS failed, Hessian reset 
    1799       4802.46   4.93604e-05       73.1128      0.7581      0.7581     2486   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1830       4802.47   5.37328e-07       52.3519      0.4238           1     2528   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -31.1687
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       4686.21     0.0261699       290.435           1           1      113   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       4852.25     0.0175201       2485.65       0.188           1      221   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       4931.92    0.00161299       475.958      0.2762      0.2762      328   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       4964.41     0.0115402       306.753           1           1      439   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       4990.19    0.00182596        153.29           1           1      548   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       5001.97      0.004941       413.754      0.4868      0.4868      662   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     655       5020.82   0.000508804       765.688   4.081e-07       0.001      761  LS failed, Hessian reset 
     699       5034.43     0.0171053       1674.56           1           1      811   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     748       5040.51   0.000144706       190.778    1.24e-06       0.001      915  LS failed, Hessian reset 
     777       5043.14   6.77194e-05       94.7324   8.288e-07       0.001      988  LS failed, Hessian reset 
     799       5043.81     0.0032043       344.557           1           1     1013   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     825       5044.39   6.09647e-05       98.2764   7.637e-07       0.001     1081  LS failed, Hessian reset 
     899       5047.14     0.0036588       209.388           1           1     1179   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     999       5051.59     0.0131089         871.8      0.2462      0.2462     1292   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1003          5052   0.000108312       122.599   3.213e-07       0.001     1341  LS failed, Hessian reset 
    1099       5057.33   0.000844129        146.09           1           1     1461   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1164       5058.14   0.000116811       174.478   8.936e-07       0.001     1589  LS failed, Hessian reset 
    1199       5058.59   0.000753165       81.5357           1           1     1631   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       5066.55     0.0255474       991.442           1           1     1746   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1319       5070.03   0.000651007       264.974   4.351e-06       0.001     1813  LS failed, Hessian reset 
    1399       5075.78   0.000135673       115.165      0.3601      0.3601     1905   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1425       5076.68   0.000135905       193.712   7.579e-07       0.001     1978  LS failed, Hessian reset 
    1459       5077.14    3.7886e-05       70.9117   4.977e-07       0.001     2057  LS failed, Hessian reset 
    1484       5077.16    3.5555e-05       72.1674   4.051e-07       0.001     2136  LS failed, Hessian reset 
    1499       5077.17   9.06549e-06       63.1235      0.8092           1     2160   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1510       5077.17   2.22525e-05       62.5432   2.961e-07       0.001     2210  LS failed, Hessian reset 
    1530       5077.17   1.91804e-07        57.916      0.1823           1     2237   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -27.0516
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       5265.02     0.0323057       711.458           1           1      114   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       5380.85     0.0269367       1103.05           1           1      226   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       5541.11     0.0312318       678.583      0.3039           1      347   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     332       5546.72   4.28907e-05       90.2665   3.884e-07       0.001      434  LS failed, Hessian reset 
     399       5550.85     0.0175223       105.478           1           1      513   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     422       5552.64   0.000245915       281.404   1.658e-06       0.001      594  LS failed, Hessian reset 
     499       5555.95   0.000369129        80.712           1           1      684   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     561       5560.11   0.000189771       359.036    3.85e-07       0.001      792  LS failed, Hessian reset 
     599       5563.17    0.00286075       352.001           1           1      833   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       5566.87    0.00380113       175.126           1           1      947   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     706       5567.31   6.58108e-05       87.8281   2.579e-07       0.001     1016  LS failed, Hessian reset 
     773       5568.51   4.66627e-05       98.3099   5.027e-07       0.001     1144  LS failed, Hessian reset 
     799       5568.65   4.40099e-05       53.5822      0.3607      0.3607     1181   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     894       5572.31   0.000368927       184.582   4.279e-06       0.001     1322  LS failed, Hessian reset 
     899       5572.62    0.00180868       90.0159      0.3421           1     1329   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     999       5573.56    0.00284963       218.352      0.6429      0.6429     1457   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099       5581.07     0.0148092       814.633           1           1     1572   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1199       5587.03   0.000331431       85.3262           1           1     1685   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       5588.33    0.00120418       86.8857           1           1     1809   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1302       5588.42   0.000148854       245.845   7.305e-07       0.001     1863  LS failed, Hessian reset 
    1399       5588.98   0.000229741       59.6016           1           1     1988   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1499       5592.68    0.00202045        283.06      0.3075      0.3075     2105   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1587       5599.15   0.000171953       218.765   2.359e-07       0.001     2291  LS failed, Hessian reset 
    1599       5600.99      0.001933       197.377           1           1     2306   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1699       5605.61   0.000317445       132.505           1           1     2419   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1799       5606.91    0.00102668       66.8966           1           1     2542   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1889       5608.02   0.000189327       128.191   2.981e-06       0.001     2692  LS failed, Hessian reset 
    1899       5608.07   0.000481109       87.0514           1           1     2702   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1930       5608.15   3.82117e-05       83.5475   2.885e-07       0.001     2788  LS failed, Hessian reset 
    1950       5608.16   1.74983e-07       48.7769     0.04762      0.8523     2816   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -42.2362
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       6032.51     0.0282767       3229.95           1           1      125   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       6321.44     0.0047487       383.977           1           1      242   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       6398.35    0.00744797       667.546           1           1      351   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       6435.03    0.00101165        486.91           1           1      462   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       6454.78     0.0379887       574.233      0.4145           1      572   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       6483.08    0.00863588       320.509           1           1      683   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       6494.22    0.00182131       150.867           1           1      796   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       6507.82     0.0113813        703.91      0.3286           1      914   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     823       6510.85    5.7098e-05       196.012   2.919e-07       0.001      984  LS failed, Hessian reset 
     899       6516.31    0.00590881       223.398           1           1     1079   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     999       6520.73   0.000305946       321.862      0.7988      0.7988     1204   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1017       6522.58   0.000449004       506.573    1.24e-06       0.001     1263  LS failed, Hessian reset 
    1099       6525.64   5.94516e-05       72.0797      0.4983      0.4983     1354   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1108        6525.7   3.57705e-05       116.105   3.351e-07       0.001     1411  LS failed, Hessian reset 
    1199        6526.6    0.00116727       187.432           1           1     1518   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       6527.38    0.00124934       89.2507           1           1     1644   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1302        6527.4   5.48647e-05        115.04   6.611e-07       0.001     1687  LS failed, Hessian reset 
    1361        6527.7   5.88633e-05       200.693    2.12e-07       0.001     1802  LS failed, Hessian reset 
    1399       6528.15   4.29497e-05        79.683      0.3305     0.03305     1850   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1499       6528.74    0.00476353       247.233           1           1     1976   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1599       6532.06    0.00350714       337.435       2.319      0.2319     2094   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1651       6534.31   5.33431e-05       167.139   4.973e-07       0.001     2197  LS failed, Hessian reset 
    1699       6534.69   3.70747e-05       131.076   3.004e-07       0.001     2295  LS failed, Hessian reset 
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1760       6534.92   2.51059e-05       67.8333   1.457e-07       0.001     2418  LS failed, Hessian reset 
    1799       6534.98   1.14291e-05       77.7745      0.3383           1     2464   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1899       6535.96   0.000114642       105.305      0.8751      0.8751     2589   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1924       6536.09   3.81244e-05       116.041   5.245e-07       0.001     2672  LS failed, Hessian reset 
    1999       6536.15    1.3054e-05       54.3662      0.2784           1     2769   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2073       6536.31   2.04638e-05        89.745    2.22e-07       0.001     2896  LS failed, Hessian reset 
    2099       6536.33   1.05374e-05       51.3195      0.3339      0.3339     2931   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2120       6536.34   8.68303e-08       55.8769    0.007093           1     2964   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -35.6581
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99        6467.5     0.0194081       4935.81      0.4104      0.4104      121   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       6725.93     0.0251917       1124.14           1           1      235   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       6843.39    0.00732954       255.366           1           1      341   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       6871.45     0.0044518       128.848           1           1      452   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       6912.17    0.00277064       742.581      0.3323           1      569   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       6930.13     0.0180923       808.367           1           1      690   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699        6940.8    0.00298338       438.176           1           1      801   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       6957.14    0.00674935       485.328           1           1      919   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899        6975.3    0.00171908       170.326           1           1     1042   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     969       6980.08     2.218e-05       86.8555   2.098e-07       0.001     1167  LS failed, Hessian reset 
     999       6980.92    0.00355218       340.809           1           1     1203   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1057       6982.03   0.000101056       165.633    6.43e-07       0.001     1308  LS failed, Hessian reset 
    1099       6982.37   6.74711e-05       66.0374           1           1     1357   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1199       6984.03   9.13525e-05       86.8377       1.684      0.1684     1482   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1256       6984.22   2.14293e-05       93.2069   3.731e-07       0.001     1588  LS failed, Hessian reset 
    1299       6984.31   7.49092e-06       62.0026           1           1     1642   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1399       6984.65    0.00029546       63.7113      0.3896           1     1763   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1499       6989.07    0.00458542       511.327          10           1     1876   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1599       6989.85   0.000179587       147.045      0.5242      0.5242     1996   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1699       6990.02   4.82941e-05       77.0681      0.7573      0.7573     2119   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1799       6990.99     0.0078841       308.509      0.1183      0.8643     2237   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1899       6991.82    0.00117742       185.029      0.9332      0.9332     2353   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1924       6991.91   6.26212e-05       209.737   4.116e-07       0.001     2428  LS failed, Hessian reset 
    1968       6991.99   3.86314e-05       113.585   6.453e-07       0.001     2524  LS failed, Hessian reset 
    1999       6992.03   2.51882e-05       63.5804      0.2002           1     2576   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2075       6992.07   1.96008e-05       80.1703   1.946e-07       0.001     2713  LS failed, Hessian reset 
    2099        6992.1   1.02182e-06        66.363     0.08022           1     2755   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2117        6992.1   5.76002e-05       55.1752   8.398e-07       0.001     2819  LS failed, Hessian reset 
    2143       6992.11   1.47933e-05       66.7187   2.396e-07       0.001     2897  LS failed, Hessian reset 
    2154       6992.11   1.12881e-06       53.4527       1.558      0.2955     2915   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -18.3728
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99        6314.2     0.0175477        1442.7           1           1      119   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       6559.44      0.164703        6230.6           1           1      229   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       6602.49    0.00194012       797.834       0.224       0.224      336   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       6627.42    0.00103155       463.351      0.1318           1      449   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       6639.68    0.00263897       176.904           1           1      564   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     555       6646.34   0.000740482       630.597   1.639e-06       0.001      670  LS failed, Hessian reset 
     599       6652.28   0.000312599       118.059      0.9681      0.9681      721   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       6658.35     0.0203467       310.663       0.336           1      837   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       6672.01     0.0133549       610.677           1           1      954   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     845       6672.85   0.000374203       147.023   6.788e-06       0.001     1064  LS failed, Hessian reset 
     884       6673.44   0.000150527        82.564    1.43e-06       0.001     1155  LS failed, Hessian reset 
     899       6673.52   6.84164e-05       85.4827      0.4032      0.4032     1171   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     923       6673.79   0.000123945       179.357   2.233e-07       0.001     1238  LS failed, Hessian reset 
     984       6674.35   4.80819e-05       111.424   3.285e-07       0.001     1352  LS failed, Hessian reset 
     999       6674.51    0.00345565       180.007           1           1     1369   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1054       6674.78   0.000579203       195.503   5.015e-06       0.001     1479  LS failed, Hessian reset 
    1099       6675.12   0.000662363        74.188      0.9469      0.9469     1536   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1199       6677.53     0.0160044       77.1806      0.2187           1     1653   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1287       6680.22   7.89172e-05       179.142   3.224e-07       0.001     1807  LS failed, Hessian reset 
    1299       6680.26   0.000110035       106.578           1           1     1822   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1306       6680.27   3.76832e-05       101.661   3.709e-07       0.001     1874  LS failed, Hessian reset 
    1322       6680.27   4.11457e-07       62.9815      0.2231      0.2231     1896   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -40.0524
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       6866.04      0.139437       4900.51           1           1      127   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       7086.81     0.0263528       1312.43      0.1847           1      235   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       7149.78     0.0155422       721.731           1           1      343   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       7170.18    0.00368012       131.296           1           1      457   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       7176.55    0.00334272       283.511       1.265      0.1265      574   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       7188.95    0.00118335       224.096           1           1      688   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       7198.24    0.00134581       314.172      0.3085      0.3085      805   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     702       7198.27   4.01595e-05       115.362   4.071e-07       0.001      872  LS failed, Hessian reset 
     744       7199.82   6.60667e-05       162.532   6.851e-07       0.001      968  LS failed, Hessian reset 
     799       7200.71   0.000355872       180.471      0.8143      0.8143     1052   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899        7204.8    0.00889985       205.142           1           1     1172   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     933       7207.47   4.54003e-05       130.888    2.11e-07       0.001     1293  LS failed, Hessian reset 
     981       7209.42   3.23774e-05       88.2021   2.053e-07       0.001     1395  LS failed, Hessian reset 
     999       7209.68   0.000489005       173.909           1           1     1417   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099       7211.15     0.0129212       216.829      0.5354           1     1535   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1199       7212.18    0.00059112       118.998           1           1     1655   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       7212.82     0.0013767        102.11           1           1     1770   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1399       7213.45   0.000113343       76.0616     0.08094           1     1898   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1499       7213.82   6.41665e-05       72.5512           1           1     2027   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1591       7214.03   3.01691e-05       75.9389    6.16e-07       0.001     2175  LS failed, Hessian reset 
    1599       7214.03   2.51877e-05       41.2182           1           1     2185   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1645       7214.16   5.01658e-06       54.2893   8.928e-08       0.001     2281  LS failed, Hessian reset 
    1666       7214.17   1.59494e-05       52.1255   2.841e-07       0.001     2349  LS failed, Hessian reset 
    1699       7214.17   6.17416e-07       64.7954      0.3107           1     2395   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1700       7214.17   5.51267e-07       69.4177           1           1     2396   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -27.5576
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       7465.85    0.00721067       3065.64      0.3003     0.03003      124   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       7582.57    0.00608465       614.935      0.9813     0.09813      241   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       7694.03    0.00413597       299.375       0.113           1      352   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       7721.63    0.00397905       187.317           1           1      475   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       7731.32    0.00313556       345.039           1           1      594   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       7749.94    0.00608973       654.097           1           1      710   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       7755.02   0.000245285       294.745      0.5454      0.5454      834   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       7758.93   0.000343115        85.263           1           1      950   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     855        7760.2   7.14781e-05       209.255   1.819e-07       0.001     1078  LS failed, Hessian reset 
     899       7761.68   9.69741e-05       66.9427           1           1     1131   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     999       7762.24   0.000159638       66.3386      0.1879           1     1248   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1016       7762.27   5.59288e-05       151.871   7.857e-07       0.001     1306  LS failed, Hessian reset 
    1045        7762.3   3.57722e-06       49.5635   6.269e-08       0.001     1377  LS failed, Hessian reset 
    1053        7762.3   9.11936e-08       51.1741     0.06398        0.23     1388   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -31.1368
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       7812.42     0.0459915       592.863           1           1      121   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       7934.57     0.0140367       583.723           1           1      230   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299        7997.3   0.000272456       240.846      0.6211      0.6211      343   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       8013.66    0.00112608       183.343           1           1      456   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       8027.47     0.0126001       172.886      0.3041           1      568   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     598        8031.9    7.8538e-05       221.858   1.763e-07       0.001      727  LS failed, Hessian reset 
     599       8031.93   0.000350466       127.705          10           1      729   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       8033.26   2.70073e-05       95.4878   2.443e-07       0.001      911  LS failed, Hessian reset 
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       8035.38    0.00121377       152.187           1           1     1033   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       8036.65     0.0134028       156.102           1           1     1160   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     949       8039.52   0.000304903       213.868   1.735e-06       0.001     1261  LS failed, Hessian reset 
     978       8040.42   3.03561e-05       61.1252   1.531e-07       0.001     1339  LS failed, Hessian reset 
     999       8040.67   0.000132946         141.4      0.4811      0.4811     1364   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1017       8040.76   7.90442e-05       151.081   1.015e-06       0.001     1434  LS failed, Hessian reset 
    1099        8041.2   0.000309053        71.308           1           1     1536   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1151       8041.34   2.60002e-05       90.2944   4.758e-07       0.001     1645  LS failed, Hessian reset 
    1199        8041.4    0.00063486       132.672           1           1     1709   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       8042.42   9.07457e-05       48.0597        2.13      0.5359     1828   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1350       8042.77   3.33668e-05       108.577    4.24e-07       0.001     1940  LS failed, Hessian reset 
    1390       8042.93   2.49545e-05       52.9379   4.459e-07       0.001     2029  LS failed, Hessian reset 
    1399       8042.94   3.50811e-06       47.0735      0.2467           1     2042   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1452       8043.07    5.9196e-06       48.5536   1.163e-07       0.001     2151  LS failed, Hessian reset 
    1499       8043.12   1.92392e-05       58.6111      0.6328      0.6328     2212   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1546       8043.13   1.77807e-06       56.3553   3.193e-08       0.001     2323  LS failed, Hessian reset 
    1558       8043.13   9.62669e-06       58.3895   1.516e-07       0.001     2382  LS failed, Hessian reset 
    1571       8043.13    3.6739e-07       52.5414      0.2374      0.2374     2399   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -38.7205
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       8248.13     0.0973434       4296.17           1           1      128   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       8439.47       0.13426       2982.07           1           1      237   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299        8556.4     0.0161573       1248.72           1           1      348   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       8608.26   0.000628346       292.132      0.9006      0.9006      459   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       8627.26    0.00410417       169.063           1           1      576   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       8654.81    0.00924663       1067.02      0.7388      0.7388      695   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       8667.27    0.00152374       242.973      0.3889      0.3889      816   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       8686.53     0.0007531       368.701      0.7616      0.7616      931   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       8701.04    0.00029555       256.832      0.3083      0.3083     1048   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     975       8703.29   0.000128307       155.303    9.47e-07       0.001     1180  LS failed, Hessian reset 
     999       8704.02   0.000407112        324.47       0.118       0.411     1211   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099       8706.37   0.000457985       146.714      0.5837      0.5837     1333   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1199       8713.24    0.00081135       222.613           1           1     1456   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1223       8713.68   0.000179955       202.465   2.052e-06       0.001     1523  LS failed, Hessian reset 
    1299       8714.39   0.000138306       67.1299     0.04347           1     1619   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1344       8714.61    4.4407e-05       142.072   5.554e-07       0.001     1727  LS failed, Hessian reset 
    1399       8715.17   0.000637985       193.359           1           1     1799   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1410       8715.36   1.92605e-05       91.5153    2.91e-07       0.001     1876  LS failed, Hessian reset 
    1499       8715.74   0.000453865       139.147      0.7483      0.7483     1992   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1512       8715.82   1.94486e-05       81.5093   1.277e-07       0.001     2057  LS failed, Hessian reset 
    1599       8716.21   0.000772128       95.7644           1           1     2170   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1699        8718.2     0.0142736        318.01           1           1     2290   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1799        8720.6   0.000345377       159.071           1           1     2405   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1825       8720.78   3.63755e-05       132.873   3.933e-07       0.001     2482  LS failed, Hessian reset 
    1863       8720.88     5.013e-05       157.929   8.233e-07       0.001     2582  LS failed, Hessian reset 
    1884       8720.89   0.000131855       109.144   1.423e-06       0.001     2642  LS failed, Hessian reset 
    1899        8720.9   2.09446e-06       39.8883       0.413       0.413     2661   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1999       8721.52    0.00206754       143.513           1           1     2782   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2000       8721.52   5.14161e-05       185.911   3.583e-07       0.001     2824  LS failed, Hessian reset 
    2035       8721.54   1.40736e-06       53.4606           1           1     2878   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -57.5928
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       9152.98     0.0279367       8319.66      0.3176           1      121   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       9590.72     0.0131907       1010.53           1           1      231   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       9766.89     0.0150256       1336.71           1           1      346   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399          9809     0.0101453       1089.95           1           1      455   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       9856.45     0.0834439       1955.59           1           1      570   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       9867.86    0.00240803       662.989           1           1      689   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       9873.88    0.00999854       586.607           1           1      803   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       9875.79    0.00104471       818.624      0.7048      0.7048      920   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       9877.64   0.000153709       154.937      0.3178      0.3178     1036   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     999       9881.98   0.000835999       699.246       0.193           1     1156   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1001       9882.07   2.40241e-05       177.527   1.398e-07       0.001     1200  LS failed, Hessian reset 
    1022       9882.38   1.42127e-05       112.698   1.189e-07       0.001     1273  LS failed, Hessian reset 
    1099       9883.11   0.000227148         248.8      0.9204      0.9204     1363   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1138       9883.56   2.38621e-05       170.531   2.385e-07       0.001     1453  LS failed, Hessian reset 
    1192       9884.02   8.18392e-06       65.0591   1.011e-07       0.001     1569  LS failed, Hessian reset 
    1199       9884.02   2.40982e-05       58.3704      0.8139      0.8139     1576   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1244       9884.43    2.6886e-05       203.556    8.78e-08       0.001     1683  LS failed, Hessian reset 
    1294       9884.58    8.1395e-06       68.0616   1.065e-07       0.001     1783  LS failed, Hessian reset 
    1299       9884.59   0.000109979       108.647           1           1     1788   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1347       9884.66   2.28971e-05        126.57   2.862e-07       0.001     1889  LS failed, Hessian reset 
    1399        9884.7   0.000327853       146.139        0.77        0.77     1956   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1491       9884.97   1.35249e-05       57.7829   2.578e-07       0.001     2111  LS failed, Hessian reset 
    1499       9884.97   3.55791e-06       72.5102      0.4415      0.4415     2120   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1505       9884.98   7.30467e-06       60.1838   9.303e-08       0.001     2165  LS failed, Hessian reset 
    1511       9884.98   3.60616e-07       48.9382      0.3069      0.9033     2175   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -75.2932
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       9931.04     0.0816429       9060.67           1           1      122   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       10292.1     0.0102366       1291.84           1           1      236   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       10386.1     0.0178082       2422.86           1           1      349   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       10492.4    0.00947808       2934.54      0.5071      0.5071      466   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       10536.1     0.0160471       1696.27           1           1      578   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599         10562     0.0037239       845.846      0.7067      0.7067      696   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       10583.9    0.00782849       1519.06           1           1      801   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       10622.9     0.0117274       464.191      0.2435      0.8412      908   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       10630.4    0.00623539       968.853      0.4217           1     1025   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     999       10638.8    0.00118793       159.121           1           1     1142   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099         10643     0.0033227       161.512           1           1     1250   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1199       10648.4     0.0048756       371.367           1           1     1363   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       10662.5     0.0363816       918.219           1           1     1473   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1399       10671.8    0.00133741       380.386           1           1     1587   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1499       10675.7   0.000398808       122.571           1           1     1704   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1599         10677   0.000807955       269.475           1           1     1811   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1699       10679.2   0.000100632       81.7613      0.4805      0.4805     1930   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1799         10680   0.000110915       570.669      0.6713      0.6713     2047   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1899       10682.1   4.61901e-05       76.8833      0.4806      0.4806     2166   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1999       10683.2   0.000140612       112.277      0.2253           1     2294   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2099       10684.4    0.00418983       270.765           1           1     2409   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2199         10686   0.000440851       143.231           1           1     2531   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2206         10686   5.78447e-05       125.347   7.154e-07       0.001     2584  LS failed, Hessian reset 
    2288       10686.3   5.28046e-06       56.5791   6.457e-08       0.001     2719  LS failed, Hessian reset 
    2299       10686.3   6.96529e-05       81.2637       0.796       0.796     2742   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2308       10686.3   4.31308e-06       46.4559   9.855e-08       0.001     2799  LS failed, Hessian reset 
    2399         10687   0.000695385       1015.26      0.7687      0.7687     2904   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2477       10688.5   2.67785e-05       174.941   2.433e-07       0.001     3037  LS failed, Hessian reset 
    2499       10688.5   7.66796e-05       122.481      0.2642     0.02642     3064   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2599       10689.5   0.000193775       68.1914           1           1     3182   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2620       10689.6   1.98151e-05       147.444   3.047e-07       0.001     3253  LS failed, Hessian reset 
    2699       10689.9   4.22901e-05       66.1422      0.7868      0.7868     3354   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2704       10689.9   1.05762e-05       68.2623   2.781e-07       0.001     3397  LS failed, Hessian reset 
    2739       10689.9   1.07598e-05       81.5875   4.975e-08       0.001     3486  LS failed, Hessian reset 
    2799         10690    0.00146664       80.0063           1           1     3567   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2843       10690.2   2.05341e-05       213.002   7.198e-08       0.001     3651  LS failed, Hessian reset 
    2899       10690.4   0.000153305       196.357           1           1     3717   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2999       10690.6   1.48924e-05       76.6417      0.2954      0.2954     3837   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    3087       10691.5   1.95687e-05        210.68   7.078e-08       0.001     3978  LS failed, Hessian reset 
    3099       10691.8   0.000393887       155.527      0.4927      0.4927     3992   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    3191       10692.7   1.74931e-05       171.752    8.39e-08       0.001     4157  LS failed, Hessian reset 
    3199       10692.7   2.75672e-05       190.413      0.3311      0.3311     4166   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    3292       10692.9   7.20235e-06       70.3667   2.218e-07       0.001     4342  LS failed, Hessian reset 
    3299       10692.9   4.46955e-05       83.6889      0.7586      0.7586     4352   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    3385         10693   7.57193e-06       53.5903   4.714e-08       0.001     4507  LS failed, Hessian reset 
    3399         10693   2.16555e-05        48.665           1           1     4526   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    3474       10693.1   1.18078e-05       99.6655   1.418e-07       0.001     4669  LS failed, Hessian reset 
    3493       10693.1   6.55945e-06       54.8591   1.268e-07       0.001     4740  LS failed, Hessian reset 
    3499       10693.1   9.56403e-07         46.34      0.3876      0.3876     4747   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    3525       10693.1    4.4678e-07        49.984   1.409e-08       0.001     4822  LS failed, Hessian reset 
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -76.7703
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       10487.6     0.0025403       1199.27           1           1      119   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       10707.5     0.0123278       1230.25           1           1      230   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       10877.1    0.00508185       1593.56           1           1      343   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       10960.2    0.00166989       442.585           1           1      454   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       10985.7    0.00273839       759.129       0.294           1      562   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       10998.4    0.00232956       344.274           1           1      679   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       11007.9    0.00624633       522.967      0.2319           1      805   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       11015.5   0.000543288       640.967      0.4082     0.04082      921   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       11018.2   0.000403134       185.232       1.314      0.1314     1035   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     951       11019.6   1.07087e-05       99.5882   6.398e-08       0.001     1143  LS failed, Hessian reset 
     999         11020   0.000516886       238.971           1           1     1196   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099       11020.7    0.00120631       272.376           1           1     1319   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1104       11020.7   3.23434e-05       200.405   4.727e-07       0.001     1368  LS failed, Hessian reset 
    1169       11021.2   7.32992e-06       69.3039   9.743e-08       0.001     1497  LS failed, Hessian reset 
    1199       11021.3   0.000153389       142.347      0.6128      0.6128     1533   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       11025.5   0.000854566       79.1312           1           1     1647   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1399       11026.8    0.00129037        99.568           1           1     1766   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1444       11027.2   8.51501e-06        87.692   1.113e-07       0.001     1868  LS failed, Hessian reset 
    1499       11027.6   0.000560088       85.0588      0.4257           1     1938   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1510       11027.9   2.59882e-05        253.44    8.87e-08       0.001     1994  LS failed, Hessian reset 
    1553       11028.2   6.57174e-06       61.9496   6.243e-08       0.001     2089  LS failed, Hessian reset 
    1599       11028.3    1.1657e-05        54.386           1           1     2153   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1699       11029.1    0.00519743        388.33           1           1     2272   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1799       11029.5   0.000112147       63.7748           1           1     2393   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1819       11029.7   1.03375e-05       102.026   8.835e-08       0.001     2456  LS failed, Hessian reset 
    1890       11029.9   6.53862e-06       80.5558   9.327e-08       0.001     2590  LS failed, Hessian reset 
    1899       11029.9   6.18135e-05       70.4546      0.3792           1     2603   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1959       11030.1   3.70533e-05       316.409   6.033e-08       0.001     2738  LS failed, Hessian reset 
    1981       11030.1   1.88755e-05       144.339   1.499e-07       0.001     2808  LS failed, Hessian reset 
    1999       11030.1   6.35231e-05       88.3708           1           1     2829   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2005       11030.2   1.81767e-05       166.803   1.116e-07       0.001     2883  LS failed, Hessian reset 
    2035       11030.2   6.59865e-06       76.4386   6.624e-08       0.001     2966  LS failed, Hessian reset 
    2043       11030.2   6.11052e-07       41.6607      0.6017      0.0152     2983   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -35.239
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       10535.8      0.104686       4017.47           1           1      119   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       10732.4    0.00516871       2666.14           1           1      228   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       10841.6     0.0008874       1202.64      0.5375      0.5375      344   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       10884.8     0.0217717       1848.95           1           1      451   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       10924.8    0.00276254       1555.98       0.889       0.889      562   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       10947.6    0.00287655        352.33           1           1      673   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     678       10951.4   4.00722e-05       248.875   1.512e-07       0.001      806  LS failed, Hessian reset 
     699       10952.2    0.00136639       352.954           1           1      830   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       10954.7   0.000271521       415.172      0.6901      0.6901      954   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       10957.5   0.000181691       752.916      0.6262      0.6262     1082   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     931         10958   2.57773e-05        151.04   6.919e-08       0.001     1164  LS failed, Hessian reset 
     999       10958.4   0.000826203       126.715           1           1     1243   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099       10961.8    0.00629785       294.892           1           1     1365   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1199       10963.8   0.000116889        81.718           1           1     1488   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1209         10964   4.02767e-05       259.976   7.261e-08       0.001     1541  LS failed, Hessian reset 
    1287         10965     1.575e-05       126.987   1.691e-07       0.001     1675  LS failed, Hessian reset 
    1299       10965.1   0.000102568       184.783           1           1     1689   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1303       10965.1   7.14833e-06       60.4598   1.054e-07       0.001     1736  LS failed, Hessian reset 
    1362       10965.4   3.85877e-05       94.8017    3.93e-07       0.001     1854  LS failed, Hessian reset 
    1399       10965.5   1.25875e-05       52.4488     0.07739           1     1898   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1499         10966   0.000847061       128.909           1           1     2014   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1599       10966.2   0.000367357       124.169           1           1     2143   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1699       10966.6     0.0042578       74.5381           1           1     2258   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1799       10966.9   1.64629e-05       71.0031      0.1175      0.1175     2382   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1829       10966.9   2.79043e-07       50.0102     0.05829           1     2420   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -39.1591
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       10680.3    0.00437173        1190.9           1           1      112   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199         10881     0.0163446       3452.18      0.4842      0.4842      229   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       10936.4     0.0138122       1857.85           1           1      341   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399       11005.1    0.00296368       1828.72      0.5667      0.5667      459   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       11033.6   8.92841e-05       185.026           1           1      578   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       11048.7    0.00049067       478.829      0.5522      0.5522      692   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       11065.8    0.00045816       166.487      0.6572      0.6572      805   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       11073.4     0.0109166       885.832           1           1      918   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     824       11083.2   9.65283e-05       647.321   1.914e-07       0.001      988  LS failed, Hessian reset 
     880       11087.4    1.8806e-05       124.849   2.019e-07       0.001     1101  LS failed, Hessian reset 
     899       11087.7    0.00017941       100.414           1           1     1122   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     978       11092.5   0.000126636       528.027   3.186e-07       0.001     1253  LS failed, Hessian reset 
     999       11094.5   0.000865883       772.526      0.2383           1     1280   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1082       11097.9   1.51475e-05       98.2482   2.657e-07       0.001     1424  LS failed, Hessian reset 
    1099         11098   0.000851983       100.828           1           1     1446   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1102         11098   3.63052e-05       155.896   2.807e-07       0.001     1497  LS failed, Hessian reset 
    1145       11098.6   7.80708e-05        198.96    1.08e-06       0.001     1607  LS failed, Hessian reset 
    1172       11098.9   1.61838e-05        108.22   2.341e-07       0.001     1676  LS failed, Hessian reset 
    1199         11099    0.00451022        136.68       2.146      0.2146     1715   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1247       11100.8   5.67265e-05       298.966   7.637e-08       0.001     1814  LS failed, Hessian reset 
    1285       11101.6   1.01765e-05       73.9558   9.686e-08       0.001     1909  LS failed, Hessian reset 
    1299       11101.8    0.00021772       208.499      0.4616      0.4616     1925   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1348       11102.1   1.17291e-05       57.8044   7.336e-08       0.001     2032  LS failed, Hessian reset 
    1399       11102.4   0.000632652        225.12           1           1     2108   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1499       11103.5    0.00621572       171.074           1           1     2226   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1599       11106.4    0.00685864       399.384           1           1     2344   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1665       11107.1   2.72952e-05       148.801   7.481e-08       0.001     2476  LS failed, Hessian reset 
    1699       11107.3   0.000712872       100.315       6.328           1     2519   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1733       11107.6   2.76835e-05       179.729   8.717e-08       0.001     2607  LS failed, Hessian reset 
    1799         11108    5.1566e-06       60.3853           1           1     2691   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1879       11108.2   4.19612e-05        141.77   5.092e-07       0.001     2830  LS failed, Hessian reset 
    1899       11108.2     0.0002195       257.795      0.6031      0.6031     2856   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1999       11108.4   1.40342e-05       125.412      0.3572      0.3572     2976   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2085       11109.2   7.40326e-05       304.166   5.975e-07       0.001     3151  LS failed, Hessian reset 
    2099       11109.5   0.000642746       122.527           1           1     3167   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2151       11109.7   1.56089e-05       109.459   1.851e-07       0.001     3283  LS failed, Hessian reset 
    2199       11109.7    1.2039e-05       45.2656      0.5473      0.5473     3336   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    2214       11109.7   8.56751e-05       95.5927   5.126e-07       0.001     3410  LS failed, Hessian reset 
    2254       11109.7   5.96609e-06       37.3981     1.5e-07       0.001     3507  LS failed, Hessian reset 
    2265       11109.7   1.94354e-07       58.9549     0.03333           1     3526   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
Initial log joint probability = -79.9284
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
      99       11448.9     0.0220876       4311.24      0.2807           1      123   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     199       11662.1     0.0123294       1430.64           1           1      228   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     299       11718.1    0.00305362       1888.48       0.153       0.153      337   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     399         11751    0.00154748       277.295      0.6112      0.6112      446   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     499       11792.1    0.00260492       622.197           1           1      562   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     599       11812.5    0.00333063       334.894      0.6999      0.6999      679   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     699       11821.3    0.00546124       303.859           1           1      791   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     799       11825.6   0.000196608       129.651           1           1      908   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     899       11831.1    0.00499126       223.506           1           1     1019   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
     999       11832.9     0.0012979       201.096       1.038      0.1038     1139   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1099       11834.5   0.000912532       248.706           1           1     1253   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1137       11834.8   3.94821e-05       101.107   5.297e-07       0.001     1351  LS failed, Hessian reset 
    1191       11835.4   4.91225e-05        65.024   4.588e-07       0.001     1464  LS failed, Hessian reset 
    1199       11835.4   0.000334031       96.5331           1           1     1472   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1299       11837.5    0.00138968       443.535      0.2777      0.2777     1596   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1347         11838   2.39388e-05       164.329   1.726e-07       0.001     1708  LS failed, Hessian reset 
    1399       11838.2   0.000120418       125.924      0.4173      0.4173     1771   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1454       11838.4   1.72799e-05       144.262   7.593e-08       0.001     1880  LS failed, Hessian reset 
    1499       11838.5   0.000169909       230.707      0.4981      0.4981     1934   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1510       11838.6   5.97351e-05        76.971   7.444e-07       0.001     1985  LS failed, Hessian reset 
    1599       11839.1     0.0013226       224.072           1           1     2097   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1619       11839.5   2.94062e-05       85.7915   5.256e-08       0.001     2158  LS failed, Hessian reset 
    1648       11839.8   1.48188e-05       133.919   8.575e-08       0.001     2240  LS failed, Hessian reset 
    1659       11839.9   8.32445e-05       273.862   1.139e-06       0.001     2295  LS failed, Hessian reset 
    1699         11840   7.79198e-05       94.0069           1           1     2347   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1799       11840.2     3.862e-06       43.5957           1           1     2475   
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes 
    1802       11840.2   3.20842e-06        63.568   6.689e-08       0.001     2528  LS failed, Hessian reset 
    1803       11840.2   1.28049e-06       45.5582           1           1     2529   
Optimization terminated normally: 
  Convergence detected: relative gradient magnitude is below tolerance
In [77]:
from fbprophet.diagnostics import performance_metrics

df_performance = performance_metrics(df_cross_validation)
df_performance.head()
Out[77]:
horizon mse rmse mae mape mdape coverage
0 36 days 4336.866586 65.854890 50.850915 0.103829 0.083400 0.328864
1 37 days 4379.928854 66.181031 51.159571 0.105126 0.083400 0.326394
2 40 days 4411.018496 66.415499 51.497435 0.105931 0.083337 0.326159
3 41 days 4469.168996 66.851844 52.000066 0.107241 0.083896 0.325218
4 42 days 4433.132431 66.581773 51.996755 0.108157 0.083582 0.323747
In [78]:
from fbprophet import plot
from fbprophet.plot import plot_cross_validation_metric

figure = plot_cross_validation_metric(df_cross_validation, metric= 'rmse')
In [79]:
prediction.tail()
Out[79]:
ds trend yhat_lower yhat_upper trend_lower trend_upper additive_terms additive_terms_lower additive_terms_upper weekly weekly_lower weekly_upper yearly yearly_lower yearly_upper multiplicative_terms multiplicative_terms_lower multiplicative_terms_upper yhat
4526 2022-02-14 1727.566766 1653.374908 1812.718314 1691.692311 1768.548764 4.003987 4.003987 4.003987 0.188017 0.188017 0.188017 3.815970 3.815970 3.815970 0.0 0.0 0.0 1731.570753
4527 2022-02-15 1728.045531 1643.997035 1813.233014 1692.070821 1769.144749 4.886363 4.886363 4.886363 0.942723 0.942723 0.942723 3.943639 3.943639 3.943639 0.0 0.0 0.0 1732.931894
4528 2022-02-16 1728.524296 1651.963782 1816.688483 1692.453056 1769.740734 5.455935 5.455935 5.455935 1.386877 1.386877 1.386877 4.069057 4.069057 4.069057 0.0 0.0 0.0 1733.980230
4529 2022-02-17 1729.003061 1649.463204 1820.876995 1692.792042 1770.336719 5.772586 5.772586 5.772586 1.595389 1.595389 1.595389 4.177197 4.177197 4.177197 0.0 0.0 0.0 1734.775647
4530 2022-02-18 1729.481825 1654.873855 1820.113164 1693.129329 1770.932704 5.247051 5.247051 5.247051 0.993757 0.993757 0.993757 4.253293 4.253293 4.253293 0.0 0.0 0.0 1734.728876